Packt Publishing website updated #47

Activity

1. tazV2 commented on May 15, 2020

   

   tazV2

   on May 15, 2020

   More actions

   man! @aojanzen isnt kidding!
   I tried 100 days of python years ago but did not expect packt changed that much!

   hello @aojanzen ,
   you have to try xpath to find right elements. i spent/straggled sometime to make it work.
   Did you see in sign in page 'https://account.packtpub.com/login?returnUrl=referrer', there will be a notification popup? you have to click that.
   Here is the code for the packt part:

   # usual stuff
   
   driver = webdriver.Firefox()
   driver.get('https://account.packtpub.com/login?returnUrl=referrer') # login page
   time.sleep(5) # let it load
   
   driver.find_element_by_xpath('//html/body/div[2]/div/div[3]/div[2]/button[1]').click() #get rid of the popup 
   
   driver.find_element_by_xpath("//html/body/app-root/div/ng-component/div/div/ng-component/div/form/div[1]/input").send_keys('username')
   driver.find_element_by_xpath('//html/body/app-root/div/ng-component/div/div/ng-component/div/form/div[2]/input').send_keys('password')
   driver.find_element_by_xpath('//html/body/app-root/div/ng-component/div/div/ng-component/div/form/button').click() # click login 
    
   

   @bbelderbos and @hobojoe1848, please update the course soon as possible

   👍React with 👍1Reacted by Dr. Andreas Janzen
2. bbelderbos commented on May 15, 2020

   

   bbelderbos

   on May 15, 2020

   Collaborator

   More actions

   Thanks @tazV2 @aojanzen, I will look at this today.

   👍React with 👍1Reacted by Tasvir Z. Khan
3. bbelderbos

   self-assigned this

   on May 15, 2020
4. bbelderbos commented on May 15, 2020

   

   bbelderbos

   on May 15, 2020

   Last edited by bbelderbos

   Collaborator

   More actions

   Sigh, not only did the design change, you cannot download books anymore, and I needed a bunch of sleeps to actually get the content to load. This works for me (loading in my Packt credentials from my environment):

   $ more script.py
   import os
   import time
   
   from selenium import webdriver
   from selenium.webdriver.common.keys import Keys
   
   USERNAME = os.environ.get('PACKT_USER')
   PASSWORD = os.environ.get('PACKT_PW')
   PACKT_LOGIN = 'https://account.packtpub.com/login'
   
   driver = webdriver.Chrome()
   driver.get(PACKT_LOGIN)
   
   # page is a bit slow
   time.sleep(3)
   
   driver.find_element_by_name('email').send_keys(USERNAME)
   driver.find_element_by_name('password').send_keys(PASSWORD + Keys.RETURN)
   
   # go to my books (make sure it's available)
   time.sleep(3)
   driver.find_element_by_link_text('My Owned Products').click()
   
   # unfortunately you cannot download titles anymore :(
   # again there is a slight delay
   time.sleep(2)
   elements = driver.find_elements_by_class_name("owned-product")
   for e in elements:
       print(e.find_element_by_class_name('title').text)
   

   Output (pagination so only first 10 results, no download links to go after):

   $ python script.py
   Mastering Python Networking - Second Edition
   Docker Cookbook - Second Edition
   Hands-On Dark Web Analysis
   Modern JavaScript Web Development Cookbook
   Hands-On Meta Learning with Python
   Data Science Algorithms in a Week - Second Edition
   Natural Language Processing with TensorFlow
   Hands-On Machine Learning for Algorithmic Trading
   Architects of Intelligence
   Hands-On Transfer Learning with Python
   

   So:

   1. I think we add a disclaimer to the README or starting video that Packt's website changed so the code won't work anymore.

   2. Maybe we can add another scraper to this repo, using something we host ourselves? In that case what would be cool?

   cc @mikeckennedy @hobojoe1848 - what do you guys think?

   👍React with 👍1Reacted by Tasvir Z. Khan
5. aojanzen commented on May 16, 2020

   

   aojanzen

   on May 16, 2020

   Author

   More actions

   Thank you, guys! I really appreciate the quick replies.