Navigating to "url", waiting until "load" - Python Playwright Issue

Hey I have code in python playwright for getting page source: This works okay, but sometimes I receive this error: I tried to add but still, these errors appear sometimes, any help, also im confused why this error appears only sometimes, what causes this error, if someone has experience please share it?

Hey I have code in python playwright for getting page source:

import json import sys import bs4 import urllib.parse from bs4 import BeautifulSoup server_proxy = urllib.parse.unquote(sys.argv[1]) link = urllib.parse.unquote(sys.argv[2]) from playwright.sync_api import sync_playwright with sync_playwright() as p: #browser = p.chromium.launch(headless = False) browser = p.chromium.launch(proxy={"server": server_proxy,'username': 'xxx',"password": 'xxx' }) context = browser.new_context(user_agent='Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36') page = context.new_page() cookie_file = open('cookies_tessco.json') cookies = json.load(cookie_file) context.add_cookies(cookies) page.goto(link) try: page.wait_for_timeout(10000) cont = page.content() print(cont) page.close() context.close() browser.close() except Exception as e: print("Error in playwright script." + page) page.close() context.close() browser.close() 

This works okay, but sometimes I receive this error:

Traceback (most recent call last): File "page_tessco.py", line 17, in <module> page.goto(link) File "/usr/local/lib/python3.9/site-packages/playwright/sync_api/_generated.py", line 5774, in goto self._sync( File "/usr/local/lib/python3.9/site-packages/playwright/_impl/_sync_base.py", line 103, in _sync return task.result() File "/usr/local/lib/python3.9/site-packages/playwright/_impl/_page.py", line 464, in goto return await self._main_frame.goto(**locals_to_params(locals())) File "/usr/local/lib/python3.9/site-packages/playwright/_impl/_frame.py", line 117, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.9/site-packages/playwright/_impl/_connection.py", line 36, in send return await self.inner_send(method, params, False) File "/usr/local/lib/python3.9/site-packages/playwright/_impl/_connection.py", line 47, in inner_send result = await callback.future playwright._impl._api_types.TimeoutError: Timeout 30000ms exceeded. =========================== logs =========================== navigating to "https://www.tessco.com/product/207882", waiting until "load" 

I tried to add

page.wait_for_timeout(10000) 

but still, these errors appear sometimes, any help, also im confused why this error appears only sometimes, what causes this error, if someone has experience please share it?

2 Answers

https://www.tessco.com/product/207882 loads quit slow. Try to extend the default timeout of 30000ms adding a timeout to page.goto(link):

page.goto(link, timeout = 0) 

With setting timeout to 0 you disable the timeout. Documentation

Alternatively, you can disable timeout with the following:

page.set_default_timeout(0) page.goto(link) 
7

Another alternative (for instances in which you sometimes experience timeouts) is to just keep retrying to load the page over and over by using a while loop that only breaks out of the loop if it is successful in its try block. The key here (and something I learned) is the fact that the continue statement in the except block doesn't return any exception, but rather retries the code within the while loop

import sleep while True: try: page.goto(link) except: sleep(<SLEEP FOR SOME AMOUNT OF SECONDS>) continue break 

The sleep is optional here, but does give your network time to recover if its a networking issue. Also if you want to have max retires (instead of infinite) you can always do:

retries = 1 max_retries = 10 while retries <= max_retries: try: page.goto(link) except: sleep(<SLEEP FOR SOME AMOUNT OF SECONDS>) retries += 1 continue break 

ncG1vNJzZmirpJawrLvVnqmfpJ%2Bse6S7zGiorp2jqbawutJobXFqZmuBdn2Op5ivoZeWwaq6xmarqGWlp7luw8Ciq6Kml2LCr8DIpWSlp5GZerHF06Gmp2Wgoa66w9GinqGsXZ7AtMHE

 Share!