java爬取并下載酷狗TOP500歌曲的方法

更新時(shí)間：2019年01月07日 10:11:17 作者：后山悟道人

這篇文章主要介紹了java爬取并下載酷狗TOP500歌曲的方法，非常具有實(shí)用價(jià)值，需要的朋友可以參考下

是這樣的，之前買車送的垃圾記錄儀不能用了，這兩天狠心買了好點(diǎn)的記錄儀，帶導(dǎo)航、音樂、藍(lán)牙、4G等功能，尋思，既然有這些功能就利用起來，用4G聽歌有點(diǎn)奢侈，就準(zhǔn)備去酷狗下點(diǎn)歌聽，居然都是需要辦會(huì)員才能下載，而且vip一月只能下載300首，我這么窮又這么摳怎么可能沖會(huì)員，于是百度搜了下怎么免費(fèi)下載，都是python爬取，雖然也會(huì)一點(diǎn)，但是電腦上沒安裝python，再安裝再研究感覺有點(diǎn)費(fèi)勁，于是就花了半小時(shí)做了這個(gè)爬蟲，技術(shù)一般，只記錄分析實(shí)現(xiàn)過程，大牛請(qǐng)繞行。其中用到了一些庫，包括：jsoup、HttpClient、net.sf.json大家可以自行去下載jar包

1、分析是否能獲得TOP500歌單

首先，打開酷狗首頁查看酷狗TOP500，說好的500首，怎么就只有22首呢，

是真的只讓看這些還是能找到其余的呢，于是我就看了下這TOP500的鏈接

https://www.kugou.com/yy/rank/home/1-8888.html?from=rank

可以看的出home后邊有個(gè)1，難道這是代表第一頁的意思？于是我就把1改成2，進(jìn)入，果然進(jìn)入了第二頁，至此可以知道我們可以在網(wǎng)頁里獲取這500首的歌單。

2.分析找到真正的mp3下載地址（這個(gè)有點(diǎn)繞）

點(diǎn)一個(gè)歌曲進(jìn)入播放頁面，使用谷歌瀏覽器的控制臺(tái)的Elements，搜一下mp3，很輕松就定位到了MP3的位置，

但是使用java訪問的時(shí)候爬取的html里卻沒有該mp3的文件地址，那么這肯定是在該頁面的位置使用了js來加載mp3，那么刷新下網(wǎng)頁，看網(wǎng)頁加載了哪些東西，加載的東西有點(diǎn)多，著重看一下js、php的請(qǐng)求，主要是看里面有沒有mp3的地址，分析細(xì)節(jié)就不用說了，

最終我在列表的

https://wwwapi.kugou.com/yy/index.php?r=play/getdata&callback=jQuery191027067069941080546_1546235744250&hash=667939C6E784265D541DEEE65AE4F2F8&album_id=0&_=1546235744251

這個(gè)請(qǐng)求里發(fā)現(xiàn)了mp3的完整地址，

"play_url": "http:\/\/fs.w.kugou.com\/201812311325\/dcf5b6449160903c6ee48035e11434bb\/G128\/M08\/02\/09\/IIcBAFrZqf2ANOadADn94ubOmaU995.mp3",

那這個(gè)js是怎么判斷是哪首歌的呢，那么只可能是hash這個(gè)參數(shù)來決定歌曲的，然后到播放頁面里找到這個(gè)hash的位置，是在下面的js里

var dataFromSmarty = [{"hash":"667939C6E784265D541DEEE65AE4F2F8","timelength":"237051","audio_name":"\u767d\u5c0f\u767d - \u6700\u7f8e\u5a5a\u793c","author_name":"\u767d\u5c0f\u767d","song_name":"\u6700\u7f8e\u5a5a\u793c","album_id":0}],//當(dāng)前頁面歌曲信息
      playType = "search_single";//當(dāng)前播放
  </script>

在去java爬取該網(wǎng)頁，查看能否爬到這個(gè)hash，果然，爬取的html里有這段js，到現(xiàn)在mp3的地址也找到了，歌單也找到了，那么下一步就用程序?qū)崿F(xiàn)就可以了。

3.java實(shí)現(xiàn)爬取酷狗mp3

先看一下爬取結(jié)果

找到了資源，程序?qū)崿F(xiàn)就好說了，其中使用到了自己寫的幾個(gè)工具類，自己整理點(diǎn)自己的工具類還是有好處的，以后遇到什么問題就沒必要重新寫了，直接拿來用就可以了。沒什么好說的了，下面直接貼出源碼

SpiderKugou.java

package com.bing.spider;

import java.io.IOException;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

import com.bing.download.FileDownload;
import com.bing.html.HtmlManage;
import com.bing.http.HttpGetConnect;

import net.sf.json.JSONObject;

public class SpiderKugou {

	public static String filePath = "F:/music/";
	public static String mp3 = "https://wwwapi.kugou.com/yy/index.php?r=play/getdata&callback=jQuery191027067069941080546_1546235744250&"
			+ "hash=HASH&album_id=0&_=TIME";
	
	public static String LINK = "https://www.kugou.com/yy/rank/home/PAGE-8888.html?from=rank";
	//"https://www.kugou.com/yy/rank/home/PAGE-23784.html?from=rank";
	
	
	public static void main(String[] args) throws IOException {
		
		for(int i = 1 ; i < 23 ; i++){
			String url = LINK.replace("PAGE", i + "");
			getTitle(url);
			//download("https://www.kugou.com/song/mfy6je5.html");
		}
	}
	
	public static String getTitle(String url) throws IOException{
		HttpGetConnect connect = new HttpGetConnect();
		String content = connect.connect(url, "utf-8");
		HtmlManage html = new HtmlManage();
		Document doc = html.manage(content);
		Element ele = doc.getElementsByClass("pc_temp_songlist").get(0);
		Elements eles = ele.getElementsByTag("li");
		for(int i = 0 ; i < eles.size() ; i++){
			Element item = eles.get(i);
			String title = item.attr("title").trim();
			String link = item.getElementsByTag("a").first().attr("href");
			
			download(link,title);
		}
		return null;
	}
	
	public static String download(String url,String name) throws IOException{
		String hash = "";
		HttpGetConnect connect = new HttpGetConnect();
		String content = connect.connect(url, "utf-8");
		HtmlManage html = new HtmlManage();
		
		String regEx = "\"hash\":\"[0-9A-Z]+\"";
		// 編譯正則表達(dá)式
		Pattern pattern = Pattern.compile(regEx);
		Matcher matcher = pattern.matcher(content);
		if (matcher.find()) {
			hash = matcher.group();
			hash = hash.replace("\"hash\":\"", "");
			hash = hash.replace("\"", "");
		}
		
		String item = mp3.replace("HASH", hash);
		item = item.replace("TIME", System.currentTimeMillis() + "");

		System.out.println(item);
		String mp = connect.connect(item, "utf-8");
		
		mp = mp.substring(mp.indexOf("(") + 1, mp.length() - 3);
		
		JSONObject json = JSONObject.fromObject(mp);
		String playUrl = json.getJSONObject("data").getString("play_url");
		

		System.out.print(playUrl + " == ");
		FileDownload down = new FileDownload();
		down.download(playUrl, filePath + name + ".mp3");
		
		System.out.println(name + "下載完成");
		return playUrl;
	}

}

HttpGetConnect.java

package com.bing.http;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.security.NoSuchAlgorithmException;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;

import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.http.HttpEntity;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.ResponseHandler;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.conn.ClientConnectionManager;
import org.apache.http.conn.scheme.Scheme;
import org.apache.http.conn.scheme.SchemeRegistry;
import org.apache.http.conn.ssl.SSLSocketFactory;
import org.apache.http.impl.client.BasicResponseHandler;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.conn.BasicHttpClientConnectionManager;
import org.apache.http.params.HttpParams; 
/**
 * @說明：
 * @author: gaoll
 * @CreateTime:2014-11-13
 * @ModifyTime:2014-11-13
 */
public class HttpGetConnect {
	
	/**
	 * 獲取html內(nèi)容
	 * @param url
	 * @param charsetName UTF-8、GB2312
	 * @return
	 * @throws IOException
	 */
	public static String connect(String url,String charsetName) throws IOException{
		BasicHttpClientConnectionManager connManager = new BasicHttpClientConnectionManager();
		
		CloseableHttpClient httpclient = HttpClients.custom()
	      .setConnectionManager(connManager)
	      .build();
		String content = "";
		
		try{
			HttpGet httpget = new HttpGet(url);
			
			RequestConfig requestConfig = RequestConfig.custom()
	        .setSocketTimeout(5000)
	        .setConnectTimeout(50000)
	        .setConnectionRequestTimeout(50000)
	        .build();
	    httpget.setConfig(requestConfig);
	    httpget.setHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
	    httpget.setHeader("Accept-Encoding", "gzip,deflate,sdch");
	    httpget.setHeader("Accept-Language", "zh-CN,zh;q=0.8");
	    httpget.setHeader("Connection", "keep-alive");
	    httpget.setHeader("Upgrade-Insecure-Requests", "1");
	    httpget.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36");
	    //httpget.setHeader("Hosts", "www.oschina.net");
	    httpget.setHeader("cache-control", "max-age=0"); 
		  
			CloseableHttpResponse response = httpclient.execute(httpget);
			
			int status = response.getStatusLine().getStatusCode();
      if (status >= 200 && status < 300) {
      	
        HttpEntity entity = response.getEntity();
        InputStream instream = entity.getContent();
        BufferedReader br = new BufferedReader(new InputStreamReader(instream,charsetName));
				StringBuffer sbf = new StringBuffer();
				String line = null;
				while ((line = br.readLine()) != null){
					sbf.append(line + "\n");
				}

				br.close();
				content = sbf.toString();
      } else {
        content = "";
      }
			
		}catch(Exception e){
			e.printStackTrace();
		}finally{
			httpclient.close();
		}
		//log.info("content is " + content);
		return content;
	}
	private static Log log = LogFactory.getLog(HttpGetConnect.class);
}

HtmlManage.java

package com.bing.html;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

import com.bing.http.HttpGetConnect;

/**
 * @說明：
 * @author: gaoll
 * @CreateTime:2014-11-13
 * @ModifyTime:2014-11-13
 */
public class HtmlManage {
	
	public Document manage(String html){
		Document doc = Jsoup.parse(html);
		return doc;
	}
	
	public Document manageDirect(String url) throws IOException{
		Document doc = Jsoup.connect( url ).get();
		return doc;
	}
	
	public List<String> manageHtmlTag(Document doc,String tag ){
		List<String> list = new ArrayList<String>();
		
		Elements elements = doc.getElementsByTag(tag);
		for(int i = 0; i < elements.size() ; i++){
			String str = elements.get(i).html();
			list.add(str);
		}
		return list;
	}
	
	public List<String> manageHtmlClass(Document doc,String clas ){
		List<String> list = new ArrayList<String>();
		
		Elements elements = doc.getElementsByClass(clas);
		for(int i = 0; i < elements.size() ; i++){
			String str = elements.get(i).html();
			list.add(str);
		}
		return list;
	}
	
	public List<String> manageHtmlKey(Document doc,String key,String value ){
		List<String> list = new ArrayList<String>();
		
		Elements elements = doc.getElementsByAttributeValue(key, value);
		for(int i = 0; i < elements.size() ; i++){
			String str = elements.get(i).html();
			list.add(str);
		}
		return list;
	}
	
	private static Log log = LogFactory.getLog(HtmlManage.class);
}

FileDownload.java

package com.bing.download;

import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileOutputStream;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;

/**
 * @說明：
 * @author: gaoll
 * @CreateTime:2014-11-20
 * @ModifyTime:2014-11-20
 */
public class FileDownload {
	
	/**
	 * 文件下載
	 * @param url 鏈接地址
	 * @param path 要保存的路徑及文件名
	 * @return
	 */
	public static boolean download(String url,String path){
		
		boolean flag = false;
		
		CloseableHttpClient httpclient = HttpClients.createDefault();
		RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(2000)
				.setConnectTimeout(2000).build();

		HttpGet get = new HttpGet(url);
		get.setConfig(requestConfig);
		
		BufferedInputStream in = null;
		BufferedOutputStream out = null;
		try{
			for(int i=0;i<3;i++){
				CloseableHttpResponse result = httpclient.execute(get);
				System.out.println(result.getStatusLine());
				if(result.getStatusLine().getStatusCode() == 200){
					in = new BufferedInputStream(result.getEntity().getContent());
					File file = new File(path);
					out = new BufferedOutputStream(new FileOutputStream(file));
					byte[] buffer = new byte[1024];
					int len = -1;
					while((len = in.read(buffer,0,1024)) > -1){
						out.write(buffer,0,len);
					}
					flag = true;
					break;
				}else if(result.getStatusLine().getStatusCode() == 500){
					continue ;
				}
			}
			
		}catch(Exception e){
			e.printStackTrace();
			flag = false;
		}finally{
			get.releaseConnection();
			try{
				if(in != null){
					in.close();
				}
				if(out != null){
					out.close();
				}
			}catch(Exception e){
				e.printStackTrace();
				flag = false;
			}
		}
		return flag;
	}

	private static Log log = LogFactory.getLog(FileDownload.class);
}

到這就結(jié)束了，有可能有些代碼沒貼全，主要代碼已經(jīng)差不多，應(yīng)該可以跑起來，多多指教。

以上就是本文的全部?jī)?nèi)容，希望對(duì)大家的學(xué)習(xí)有所幫助，也希望大家多多支持腳本之家。

您可能感興趣的文章: