Get Element By Xpath Loop Error Selenium Python

Get Element By Xpath Loop Error Selenium Python



I'm trying to make a web scraper for Pinterest. I'm able to get almost all the data, but each pin has a button called "see more" which generates: 'board name' and 'author name' data.



Logic:



Button Click Loop Code:


moreButtons = driver.find_elements_by_xpath('//button[@data-test-id="seemoretoggle"]')
for moreBtn in moreButtons:
moreBtn.click()

source_data = driver.page_source



Get Board Name Code


# Pin Length - Total Pins
total_pins =
total_pins = driver.find_elements_by_class_name("Grid__Item")

# Pin Board Names
i = 1
while i <= len(total_pins):
temp_xpath = "/html/body/div[1]/div[1]/div[1]/div/div/div/div/div[1]/div/div/div/div[" + str(i) + "]/div/div/div[2]/div[2]/h4/a[1]"
temp = driver.find_element_by_xpath(temp_xpath)
#pin_Board_Names.append(temp)
print(temp.text)
i += 1



Kind Of Works.. Partially..


Just old
Tiny House interior
SimpleLivingMama.com
Traceback (most recent call last):
File "scrape.py", line 109, in <module>
main()
File "scrape.py", line 106, in main
grab(args.url, args.fname)
File "scrape.py", line 91, in grab
temp = driver.find_element_by_xpath(temp_xpath)
File "C:Usersda74AppDataRoamingPythonPython36site-packagesseleniumwebdriverremotewebdriver.py", line 393, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "C:Usersda74AppDataRoamingPythonPython36site-packagesseleniumwebdriverremotewebdriver.py", line 966, in find_element
'value': value})['value']
File "C:Usersda74AppDataRoamingPythonPython36site-packagesseleniumwebdriverremotewebdriver.py", line 320, in execute
self.error_handler.check_response(response)
File "C:Usersda74AppDataRoamingPythonPython36site-packagesseleniumwebdriverremoteerrorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: "errorMessage":"Unable to find element with xpath '/html/body/div[1]/div[1]/div[1]/div/div/div/div/div[1]/div/div/div/div[4]/div/div/div[2]/div[2]/h4/a[1]'","request":"headers":"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"187","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:57743","User-Agent":"selenium/3.13.0 (python windows)","httpVersion":"1.1","method":"POST","post":""using": "xpath", "value": "/html/body/div[1]/div[1]/div[1]/div/div/div/div/div[1]/div/div/div/div[4]/div/div/div[2]/div[2]/h4/a[1]", "sessionId": "a8cdaa10-a2d3-11e8-86db-a3b39599a684"","url":"/element","urlParsed":"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":,"chunks":["element"],"urlOriginal":"/session/a8cdaa10-a2d3-11e8-86db-a3b39599a684/element"
Screenshot: available via screen



It found 3 board names for me, but then it ends with errors. I tried to edit loop and button click, but they all seem to work.
Does anyone know what is causing it or maybe suggestions to explore?



Edit 1: Saw the error says cannot find element by xpath. But the element is there on the webpage.



Edit 2: Added try:except to check. Here the code:


try:
temp = driver.find_element_by_xpath(temp_xpath)
except:
print('no element at pin number: ' + str(i))



with output:


Just old
Tiny House interior
SimpleLivingMama.com
no element at pin number: 4
SimpleLivingMama.com
Books for Pre-Schoolers
Stuff to Try
Baby & Toddler Milestones
Toys For Boys & Girls
House
OT
Make Extra Money
Shoes
Old photos
Crafts
for baby
There's A Book About That
Geek
Real DIY
Recycle & Repurpose
Crafts
Preschool Activities
Wild West Project
#BossMoms
no element at pin number: 24
#BossMoms
Crazy for DIY
Money Saving Tips
Painting Furniture
The home I want
screen door ideas
DIY Home
Little girl rooms
Container Home Desing
Bentley Joseph Adams
some truth bombs
New house!
Advice and Wisdom-Words
no element at pin number: 37
Advice and Wisdom-Words
House ideas
Houses
no element at pin number: 40
Houses
no element at pin number: 41
Houses
Fine Motor Activities for Kids
crafts
decorating ideas
mama
Barn Homes
For the Home
no element at pin number: 48
For the Home



Checked the pin number where can't find output, but the board name is there on webpage.



Edit 3: Noticed that just after pin number 47, it always says no element found. No matter how big the list is. Also checked that all buttons xpaths are there in moreButtons and they're valid..



Thanks for help in advance





how many elements can you visually see on the webpage?
– AnkDasCo
yesterday






all elements.. I'm using scroll script that works.
– SimpleCoder
yesterday





Are you waiting (programmatically) for the element to load and identify if it is there or not in your code?
– AnkDasCo
yesterday





I think it does that when .click() is done in code. Here is image with button and div it displays when pressed: imgur.com/a/DPVZuAH
– SimpleCoder
yesterday





Can you use WebDriverWait on the element when it raises an exception and let it try again. Let me know how it goes. I think this happens when the element is not visible to selenium
– AnkDasCo
yesterday




1 Answer
1



As helped by @AnkDasCo in the comments, found a solution to it. There were 2 problems here:



xpaths
The following are 2 xpaths for the same item:



As we notice, the last /div in both is different.



Working Code


driver = webdriver.PhantomJS(executable_path='phantomjs.exe')
print("Ghost Headless Driver Invoked")
# driver.implicitly_wait(5) # if element not found, wait for (seconds) before next operation
driver.get(url) # grab the url

# Scrolling till the end of page
print("Started Scrolling ... ")
match=True # change to 'False' for making this work..
lenOfPage = driver.execute_script("window.scrollTo(0, document.body.scrollHeight);var lenOfPage=document.body.scrollHeight;return lenOfPage;")
while(match==False):
lastCount = lenOfPage
lenOfPage = driver.execute_script("window.scrollTo(0, document.body.scrollHeight);var lenOfPage=document.body.scrollHeight;return lenOfPage;")
if lastCount==lenOfPage:
match=True

source_data = driver.page_source # page source code as html

# Get all pins , number of pins collected
total_pins =
try:
total_pins = driver.find_elements_by_class_name("Grid__Item")
except:
print("Unable to load pins")
print("Total Pins: " + str(len(total_pins)))

# get number of 'see more' buttons collected - for error checking
moreButtons = driver.find_elements_by_xpath('//button[@data-test-id="seemoretoggle"]')
print("Dynamic Elements: " + str(len(moreButtons)))
print("Display: Dynamic Elements ... ")

# clicking all 'See More' buttons
i = 0
while i <= (len(moreButtons) - 1):
moreButtons[i].click()
i += 1

# Pin Board Names
print("Extracting Board Names ... ")
i = 1
successful = False # for checking success of try | else not working
while i <= len(total_pins):
try:
temp_xpath = "/html/body/div[1]/div[1]/div[1]/div/div/div/div/div[1]/div/div/div/div[" + str(i) + "]/div/div/div[2]/div[2]/h4/a[1]"
temp = driver.find_element_by_xpath(temp_xpath)
pin_Board_Names.append(temp.text)
# print("Board_No: " + str(i) + " > " + temp.text)
successful = True
except:
temp_xpath = "/html/body/div[1]/div[1]/div[1]/div/div/div/div/div[1]/div/div/div/div[" + str(i) + "]/div/div/div[2]/div/h4/a[1]"
temp = driver.find_element_by_xpath(temp_xpath)
pin_Board_Names.append(temp.text)
# print("Board_No: " + str(i) + " > " + temp.text)
successful = True
if successful == False:
print("Board_No: " + str(i) + " not found!")
i += 1

# quit driver
driver.quit()






By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

-

Popular posts from this blog

Help:Category

Image
Help:Category From Wikipedia, the free encyclopedia Jump to navigation Jump to search For guidelines on the use of categories in Wikipedia, see Wikipedia:Categorization. For a basic readers' guide see Help:Categories. For quick answers, see the readers' FAQ or the editors' FAQ on categories. For a list of categories, see Special:Categories. This is an information page. It describes the editing community's established practice on some aspect or aspects of Wikipedia's norms and customs. It is not one of Wikipedia's policies or guidelines, as it has not been thoroughly vetted by the community. Shortcut H:CAT Wikipedia data structure Namespaces Subject namespaces Talk namespaces 0 (Main/Article) Talk 1 2 User User talk 3 4 Wikipedia Wikipedia talk 5 6 File File talk 7 8 MediaWiki MediaWiki talk 9 10 Template Template talk 11 12 Help Help talk 13 14 Category Category talk 15 100 Portal Portal talk 101 108 Book Book tal...
Read more

How can temperature be calculated given relative humidity and dew point?

Image
How can temperature be calculated given relative humidity and dew point? While looking into the better indicator of how miserable it feels outside, either dew point or relative humidity, I came across this statement: The optimum combination for human comfort is a dewpoint of about 60 F and a RH of between 50 and 70% (this would put the temperature at about 75 F). Source: http://www.theweatherprediction.com/habyhints/190/ This led me to this calculator that will calculate the temperature given relative humidity and dew point - for example a dew point of 70 degrees F and a relative humidity of 90% results in a temperature of 73.11 degrees F. The web site for this calculator says the values are based on the August-Roche-Magnus approximation and gives the following equation to calculate temperature: $T =243.04 Large fracfrac17.625 TD243.04+TD-lnleft(fracRH100right)17.625+lnleft(fracRH100right)- frac17.625 TD243.04+TD$ Given the equation I'm still having a hard time figuring out...
Read more

I have a recursive function to validate tree graph and need a return condition

Image
I have a recursive function to validate tree graph and need a return condition I have a tree graph. Each node has attribute 'amount'. The rule governing the attribute value of root is this, The root 'amount' value is the sum of the 'amount' attribute of each of it's children. This continues to the last node with children. In other words this tree's attributes are unlike a sum tree, because the root node is not the sum of each node in the tree. Here is a toy example as graph G: nodedict = 'apples': 'amount': 5.0, 'bananas': 'amount': 10.0, 'tomato': 'amount': 50.0, 'total_fruits': 'amount': 15.0, 'total_vegetables': 'amount': 9.0, 'carrot': 'amount': 3.0, 'squash': 'amount': 6.0, 'total_fruits_and_vegetables': 'amount': 74.0 edgelist = [('total_fruits', 'apples'), ('total_fruits', 'bananas...
Read more