ó y]Rc@s´ddlZddlZddlZddlZddlZdZyddlmZWn#e k r{Z ddlmZnXde fd„ƒYZ dej fd„ƒYZdefd„ƒYZd dd „ƒYZd ejfd „ƒYZdd lmZmZmZmZmZmZmZddlZddlZddlZddlZddlZdejfd„ƒYZ dej!fd„ƒYZ!da"d„Z#d„Z$d„Z%dS(iÿÿÿÿN(tStringIOt RangeErrorcBseZdZRS(s6Error raised when an unsatisfiable range is requested.(t__name__t __module__t__doc__(((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR"stHTTPRangeHandlercBs eZdZd„Zd„ZRS(s»Handler that enables HTTP Range headers. This was extremely simple. The Range header is a HTTP feature to begin with so all this class does is tell urllib2 that the "206 Partial Content" response from the HTTP server is what we expected. Example: import urllib2 import byterange range_handler = range.HTTPRangeHandler() opener = urllib2.build_opener(range_handler) # install it urllib2.install_opener(opener) # create Request and set Range header req = urllib2.Request('http://www.python.org/') req.header['Range'] = 'bytes=30-50' f = urllib2.urlopen(req) cCs1tj|||jƒƒ}||_||_|S(N(turllibt addinfourlt get_full_urltcodetmsg(tselftreqtfpR R thdrstr((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pythttp_error_206>s  cCstddƒ‚dS(Ni sRequested Range Not Satisfiable(R(R R R R R R((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pythttp_error_416Es(RRRRR(((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR&s tHTTPSRangeHandlercBs eZdZd„Zd„ZRS(s! Range Header support for HTTPS. cCs|j|||||ƒS(N(R(R R R R R R((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pythttps_error_206LscCs|j|||||ƒdS(N(thttps_error_416(R R R R R R((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pyROs(RRRRR(((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pyRIs tRangeableFileObjectcBsheZdZd„Zd„Zd„Zdd„Zdd„Zdd„Zd „Z d „Z d „Z RS( s"File object wrapper to enable raw range handling. This was implemented primarilary for handling range specifications for file:// urls. This object effectively makes a file object look like it consists only of a range of bytes in the stream. Examples: # expose 10 bytes, starting at byte position 20, from # /etc/aliases. >>> fo = RangeableFileObject(file('/etc/passwd', 'r'), (20,30)) # seek seeks within the range (to position 23 in this case) >>> fo.seek(3) # tell tells where your at _within the range_ (position 3 in # this case) >>> fo.tell() # read EOFs if an attempt is made to read past the last # byte in the range. the following will return only 7 bytes. >>> fo.read(30) cCs>||_t|ƒ\|_|_d|_|j|jƒdS(sCreate a RangeableFileObject. fo -- a file like object. only the read() method need be supported but supporting an optimized seek() is preferable. rangetup -- a (firstbyte,lastbyte) tuple specifying the range to work over. The file object provided is assumed to be at byte offset 0. iN(tfotrange_tuple_normalizet firstbytetlastbytetrealpost_do_seek(R Rtrangetup((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt__init__gs  cCs/t|j|ƒr"t|j|ƒSt|‚dS(s«This effectively allows us to wrap at the instance level. Any attribute not found in _this_ object will be searched for in self.fo. This includes methods.N(thasattrRtgetattrtAttributeError(R tname((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pyt __getattr__uscCs|j|jS(s5Return the position within the range. This is different from fo.seek in that position 0 is the first byte position of the range tuple. For example, if this object was created with a range tuple of (500,899), tell() will return 0 when at byte position 500 of the file. (RR(R ((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pyttell}sicCs|dkr|j|}n7|dkr8|j|}n|dkrStdƒ‚n|jrw||jkrw|j}n|j||jƒdS(seSeek within the byte range. Positioning is identical to that described under tell(). iiis$seek from end of file not supported.N(RRtIOErrorRR(R toffsettwhencet realoffset((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pytseek†s    iÿÿÿÿcCs:|j|ƒ}|jj|ƒ}|jt|ƒ7_|S(s`Read within the range. This method will limit the size read based on the range. (t_calc_read_sizeRtreadRtlen(R tsizetrslt((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR*™scCs:|j|ƒ}|jj|ƒ}|jt|ƒ7_|S(sfRead lines within the range. This method will limit the size read based on the range. (R)RtreadlineRR+(R R,R-((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR.¢scCsX|jrT|dkrA|j||jkrQ|j|j}qQqT|j|j}n|S(sSHandles calculating the amount of data to read based on the range. iÿÿÿÿ(RR(R R,((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR)«s   cCsLt|jdƒs"|j|ƒn|jj|j|ƒ|j|7_dS(sSeek based on whether wrapped object supports seek(). offset is relative to the current position (self.realpos). R(N(RRt_poor_mans_seekR(R(R R%((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR·scCs€d}d}xm||kr{|||kr8||}n|jj|ƒ}t|ƒ|krntddƒ‚n||7}qWdS(s¯Seek by calling the wrapped file objects read() method. This is used for file like objects that do not have native seek support. The wrapped objects read() method is called to manually seek to the desired position. offset -- read this number of bytes from the wrapped file object. raise RangeError if we encounter EOF before reaching the specified offset. iii sRequested Range Not SatisfiableN(RR*R+R(R R%tpostbufsizetbuf((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR/Âs  ( RRRRR"R#R(R*R.R)RR/(((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pyRRs    tFileRangeHandlercBseZdZd„ZRS(s~FileHandler subclass that adds Range support. This class handles Range headers exactly like an HTTP server would. c CsÃddl}ddl}|jƒ}|jƒ}tj|ƒ}tj|ƒ}|tj}t j |tj ƒ} |j |ƒd} |rÕtj |ƒ\}} | sÀtj|ƒ|jƒkrÕtjdƒ‚qÕnt|dƒ} |jjddƒ} t| ƒ} | r„| \}}|dkr,|}n|dksP||ksP||krbtddƒ‚n||}t| ||fƒ} n|jtd | p™d || fƒƒ}tj| |d |ƒS( Niÿÿÿÿisfile not on local hosttrbtRangeti sRequested Range Not Satisfiables6Content-Type: %s Content-Length: %d Last-modified: %s s text/plainsfile:(t mimetypest mimetoolstget_hostt get_selectorRt url2pathnametoststattST_SIZEtrfc822t formatdatetST_MTIMEt guess_typet splitporttsockett gethostbynamet get_namesturllib2tURLErrortopentheaderstgettNonetrange_header_to_tupleRRtMessageRR(R R R7R8thosttfilet localfiletstatsR,tmodifiedtmtypetportRtbrangetfbtlbRJ((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pytopen_local_fileÛs8     !   $  (RRRRY(((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR3Ös(RCt splitusert splitpasswdt splitattrtunquotet addclosehookRtFTPRangeHandlercBseZd„Zd„ZRS(cCs¦|jƒ}|std‚nt|ƒ\}}|dkrHtj}n t|ƒ}t|ƒ\}}|rt|ƒ\}}nd}t |ƒ}t |pŸdƒ}t |p±dƒ}yt j |ƒ}Wn%t j k rñ}t j|ƒ‚nXt|jƒƒ\}}|jdƒ} tt | ƒ} | d | d} } | r[| d r[| d} ny|j||||| ƒ} | r…dpˆd } xM|D]E} t| ƒ\} }| jƒd kr’|dkr’|jƒ} q’q’Wd}t|jjddƒƒ}|r&|\}}|dkr&|}q&n| j| | |ƒ\}}|rÜ|\}}|dkrº|dksw|dkr‰tddƒ‚n|}||}|dkrÙtddƒ‚qÙqÜ||}t|d|fƒ}nd}tj|jƒƒd}|r|d|7}n|dk r;|dkr;|d|7}nt|ƒ}t j!|ƒ}t"|||jƒƒSWn2tj#k r¡}td|ft$j%ƒd‚nXdS(Ns ftp errors no host givenR6t/iÿÿÿÿiitItDttypetatAtitdR5i s@Requested Range Not Satisfiable due to unobtainable file length.sRequested Range Not SatisfiablesContent-Type: %s sContent-Length: %d i(s ftp errors no host given(RdReRfRaRgRb(&R9R$RCRLtftplibtFTP_PORTtintRZR[R]RDREterrorRGRHR\R:tsplittmapt connect_ftptlowertupperRMRJRKtretrfileRRR7RBRRR8RNRt all_errorstsystexc_info(R R RORUtusertpasswdR tpathtattrstdirsRPtfwRctattrtvaluetrestt range_tupRWRXR tretrlenRJRTtsf((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pytftp_open sz                 cCst|||||ƒ}|S(N(t ftpwrapper(R RuRvRORURyRz((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pyRnXs(RRRRn(((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR_ s NR‚cBseZdd„ZRS(c Cs|jƒ|dkr%d}d}nd|}d}y|jj|ƒWn.tjk ry|jƒ|jj|ƒnXd}|r§| r§y|jj|ƒWn2tjk rÕ}t d|ft j ƒd‚nX|jj|ƒy#d |}|jj ||ƒ}Wq§tjk r£}t |ƒd d krk|j||ƒ\}} t||d fƒ}|| fSt |ƒd d kr¤t d|ft j ƒd‚q¤q§Xn|së|jjdƒ|rÐd|}nd}|jj |ƒ}nd|_t|djdƒ|jƒ|dfS(NRgRbsTYPE AisTYPE is ftp errorisRETR it501R6t550sLIST tLISTR4(RgRb(t endtransfertftptvoidcmdRhRrtinitRLtnlstt error_permR$RsRtt ntransfercmdtstrRqRtbusyR^tmakefile( R RPRcR}tcmdtisdirtconntreasonR R((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pyRqasJ         &  N(RRRLRq(((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR‚\scCsš|dkrdStdkr:ddl}|jdƒantj|ƒ}|r–t|jddƒƒ}|r’|dr’|d|ddf}n|SdS(sØGet a (firstbyte,lastbyte) tuple from a Range header value. Range headers have the form "bytes=-". This function pulls the firstbyte and lastbyte values and returns a (firstbyte,lastbyte) tuple. If lastbyte is not specified in the header value, it is returned as an empty string in the tuple. Return None if range_header is None Return () if range_header does not conform to the range spec pattern. iÿÿÿÿNs^bytes=(\d{1,})-(\d*)iii((RLt_rangeretretcompiletmatchRtgroup(t range_headerR•R—ttup((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pyRM‘s   cCsS|dkrdSt|ƒ}|rO|drG|d|ddf}nd|SdS(s•Convert a range tuple to a Range header value. Return a string of the form "bytes=-" or None if no range is needed. iis bytes=%s-%sN(RLR(R~((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pytrange_tuple_to_header¬s   cCsÛ|dkrdS|d}|dkr/d}n t|ƒ}y|d}Wntk rbd}n1X|dkrxd}n|dkr“t|ƒ}n||fdkr©dS||krÑtdd||fƒ‚n||fS( s7Normalize a (first_byte,last_byte) range tuple. Return a tuple whose first element is guaranteed to be an int and whose second element will be '' (meaning: the last byte) or an int. Finally, return None if the normalized tuple == (0,'') as that is equivalent to retrieving the entire file. iR6ii sInvalid byte range: %s-%sN(NR6(iR6(RLRjt IndexErrorR(R~RWRX((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pyR¸s&        ((&R<R=RRGR?RLtDEBUGt cStringIORt ImportErrorR R$Rt BaseHandlerRRRt FileHandlerR3RCRZR[R\R]R^RRhRDRsR7R8t FTPHandlerR_R‚R”RMR›R(((s8/usr/lib/python2.7/site-packages/urlgrabber/byterange.pyts4     # „+4     S4