Or (bash/zsh): curl "http://pastebin.com/raw.php?i={Kc9ng18h,vCMndK2L,JdQkuYwG,f...

ricardobeat · on May 8, 2012

    curl "http://pastebin.com/raw.php?i={Kc9ng18h,vCMndK2L,JdQkuYwG,fw43srjY,jv4LBjPX}" | sort -u > twitterpw.txt

dredmorbius · on May 9, 2012

Sure, but that doesn't tell you how many dupes were in the original list (unless you were to separately keep a linecount). Hrm ... does pv let you do that?

Hrm ... No, but process substitution does:

    curl "http://pastebin.com/raw.php?i={Kc9ng18h,vCMndK2L,JdQkuYwG,fw43srjY,jv4LBjPX}" | tee >(sort -u >twitterpw.txt) | wc -l
    [1/5]: http://pastebin.com/raw.php?i=Kc9ng18h --> <stdout>
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  261k    0  261k    0     0   788k      0 --:--:-- --:--:-- --:--:--  953k
    
    [2/5]: http://pastebin.com/raw.php?i=vCMndK2L --> <stdout>
    100  434k    0  434k    0     0  1630k      0 --:--:-- --:--:-- --:--:-- 1630k
    
    [3/5]: http://pastebin.com/raw.php?i=JdQkuYwG --> <stdout>
    100  349k    0  349k    0     0  1526k      0 --:--:-- --:--:-- --:--:-- 7441k
    
    [4/5]: http://pastebin.com/raw.php?i=fw43srjY --> <stdout>
    100  367k    0  367k    0     0   897k      0 --:--:-- --:--:-- --:--:--  897k
    
    [5/5]: http://pastebin.com/raw.php?i=jv4LBjPX --> <stdout>
    100  291k    0  291k    0     0  1638k      0 --:--:-- --:--:-- --:--:-- 1638k
    58978

And what did we actually output?

    $ wc -l twitterpw.txt 
    37001 twitterpw.txt