webdriver调试报错,如何学好python编制程序呢

Python爬虫很强大,在爬虫里如何自动操控浏览器呢?我们知道在浩瀚的搜素引擎中,有成千上百亿只爬虫,每天往来于互联网之中,那么如此强大的互联网中爬虫是如何识别浏览器的呢,又是如何抓取数据的呢?

selenium webdriver调试报错 Message: Failed to convert data to an object?
1、selenium webdriver调试报错 Message: Failed to convert data to an
object

前言

概述:

2、代码

   
前面都是点点滴滴的介绍selenium的一些api使用方法,那么selenium的api到底有多少呢?本篇就叫大家如何去查看selenium
api,不求人,无需伸手找人要,在自己电脑就有。

python通过selenium爬取数据是很多突破封锁的有效途径。但在使用selenium中会遇到很多问题,本文就通过一问一答的形式来通熟易懂的普及如何通过selenium执行javascript程序,进而获取动态执行后的网页。如果你喜欢,欢迎转发本文。

coding =utf-8

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
import os

browser=webdriver.Firefox() #打开火狐浏览器

browser.maximize_window() #最大化浏览器

browser.implicitly_wait #智能等待时间

browser.get(“”)
#访问

browser.find_element_by_id(“btnSubmit”).click() #点击系统管理

browser.find_element_by_link_text.click() #点击组织管理

browser.find_element_by_link_text.click() #点击用户管理

browser.find_element_by_xpath(“//a[contains,’新增’)]”).click()
#点击用户管理界面新增按钮

browser.quit() #关闭浏览器

3、报错信息
C:\Users\yinxinyu\AppData\Local\Programs\Python\Python37\python.exe
D:/Work/Project/python/text3.py
Traceback (most recent call last):
File “D:/Work/Project/python/text3.py”, line 23, in
browser.find_element_by_xpath(“//a[contains,’新增’)]”).click()
File
“C:\Users\yinxinyu\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\webdriver.py”,
line 394, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File
“C:\Users\yinxinyu\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\webdriver.py”,
line 978, in find_element
‘value’: value})[‘value’]
File
“C:\Users\yinxinyu\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\webdriver.py”,
line 321, in execute
self.error_handler.check_response
File
“C:\Users\yinxinyu\AppData\Local\Programs\Python\Python37\lib\site-packages\selenium\webdriver\remote\errorhandler.py”,
line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Failed to
convert data to an object

4、尝试过的办法:更换定位方式class以及Xpath都不行

5、报错截图
可靠的线上网赌网站 1

   
pydoc是Python自带的模块,主要用于从python模块中自动生成文档,这些文档可以基于文本呈现的、也可以生成WEB
页面的,还可以在服务器上以浏览器的方式呈现!

可靠的线上网赌网站 2

一、pydoc

Python爬虫采集数据容易吗 怎么样学好python编程呢

    1.到底什么是pydoc? ,这个是准确的解释:Documentation generator and
online help
system. pydoc是Python自带的模块,主要用于从python模块中自动生成文档,这些文档可以基于文本呈现的、也可以生成WEB
页面的,还可以在服务器上以浏览器的方式呈现!简而言之,就是帮你从代码和注释自动生成文档的工具。

python爬虫编程:用selenium执行javascript出错了,该咋改?

   
2.举个栗子,我需要查看python里面open函数的功能和语法,打开cmd,输入:python
-m pydoc open

问题:

    3.-m参数:python以脚本方法运行模块

小明开始学习python爬虫编程了,仿佛整个互联网的数据都快被他纳入囊中了。今天,他又试图完成一个高难度动作,他想让selenium中抓取到以下HTML后,并自动执行js脚本,模仿鼠标自动执行一个点击动作。但令他很失望的是,居然,居然,没用!

>>python -m pydoc open

Nyaralego,Sikonge,Ab-Titchazand11otherslikethis.

 可靠的线上网赌网站 3

这是他执行的代码。

那么问题来了,这个是已经知道有这个函数,去查看它的功能,selenium里面不知道到底有多少个函数或方法,那如何查看呢?

browser.execute_script(“document.getElement(By.xpath(“//div[@class=’vbseo_liked’]/a[contains(@onclick,’returnvbseoui.others_click]”).click

二、启动server

它没用,没有反应。究竟做错了什么?

    1.打开cmd命令行,输入:python -m pydoc -p 6666

Python大大的答案:

    2.-p参数:这个表示在本机上启动服务

要点回答:

    3.6666参数:这个是服务端口号,随意设置

使用selenium查找元素并将其传递execute_script()给单击:

可靠的线上网赌网站 4

link=browser.find_element_by_xpath(‘//div[@]/a[contains(@onclick,”returnvbseoui.others_click]’)browser.execute_script(‘arguments[0].click

打开后,界面会出现一个地址:

如果要从头解决这问题,那么以下就是需要了解它的一系列事情:如何使用JavaScript模拟点击?

三、浏览器查看文档

这就是我做的东西。这很简单,但它有效:

    1.在浏览器输入:

functioneventFire{if(el.fireEvent){el.fireEvent(‘on’+etype);}else{varevObj=document.createEvent;evObj.initEvent(etype,true,false);el.dispatchEvent;}}

    2.Built-in Moudles :这个是python自带的模块

用法:

可靠的线上网赌网站 5

eventFire(document.getElementById(‘mytest1′),’click’);如何在Python里进行模拟点击呢?首先制定一个自定义的预期条件,等待元素被“执行”:

 

classwait_for_text_not_to_end_with:def__init__(self,locator,text):self.locator=locatorself.text=textdef__call__(self,driver):try:element_text=EC._find_element(driver,self.locator).text.strip()returnnotelement_text.endswith(self.text)exceptStaleElementReferenceException:returnFalse

四、webdriver API

定义完毕后,如何在程序里调用这个类呢?看看以下代码:

    1.找到这个路径:python2.7\lib\site-packages,点开selenium

fromseleniumimportwebdriverfromselenium.common.exceptionsimportStaleElementReferenceExceptionfromselenium.webdriver.common.byimportByfromselenium.webdriver.support.uiimportWebDriverWaitfromselenium.webdriver.supportimportexpected_conditionsasECclasswait_for_text_not_to_end_with:def__init__(self,locator,text):self.locator=locatorself.text=textdef__call__(self,driver):try:element_text=EC._find_element(driver,self.locator).text.strip()returnnotelement_text.endswith(self.text)exceptStaleElementReferenceException:returnFalsebrowser=webdriver.PhantomJS()browser.maximize_window()browser.get(“)

   
2.打开的selenium>webdriver>firefox>webdriver,最终路径:

看,如何在python里通过selenium来爬取数据就是这么简单。要点掌握好,开始编制自己的爬虫吧。

    3.最终看到的这些就是selenium的webdriver API帮助文档啦

用爬虫采集数据就是这么简单,如果你对python编程感兴趣,那就好好的学习下吧,用爬虫采集数据一点都不难,对于这个你开始学习了吗?

可靠的线上网赌网站 6

 

add_cookie(self, cookie_dict)
Adds a cookie to your current session.

:Args:

  • cookie_dict: A dictionary object, with required keys – “name” and
    “value”;
    optional keys – “path”, “domain”, “secure”, “expiry”

Usage:
driver.add_cookie({‘name’ : ‘foo’, ‘value’ : ‘bar’})
driver.add_cookie({‘name’ : ‘foo’, ‘value’ : ‘bar’, ‘path’ : ‘/’})
driver.add_cookie({‘name’ : ‘foo’, ‘value’ : ‘bar’, ‘path’ : ‘/’,
‘secure’:True})

back(self)
Goes one step backward in the browser history.

:Usage:
driver.back()

close(self)
Closes the current window.

:Usage:
driver.close()

create_web_element(self, element_id)
Creates a web element with the specified element_id.

delete_all_cookies(self)
Delete all cookies in the scope of the session.

:Usage:
driver.delete_all_cookies()

delete_cookie(self, name)
Deletes a single cookie with the given name.

:Usage:
driver.delete_cookie(‘my_cookie’)

execute(self, driver_command, params=None)
Sends a command to be executed by a command.CommandExecutor.

:Args:

  • driver_command: The name of the command to execute as a string.
  • params: A dictionary of named parameters to send with the command.

:Returns:
The command’s JSON response loaded into a dictionary object.

execute_async_script(self, script, *args)
Asynchronously Executes JavaScript in the current window/frame.

:Args:

  • script: The JavaScript to execute.
  • \*args: Any applicable arguments for your JavaScript.

:Usage:
driver.execute_async_script(‘document.title’)

execute_script(self, script, *args)
Synchronously Executes JavaScript in the current window/frame.

:Args:

  • script: The JavaScript to execute.
  • \*args: Any applicable arguments for your JavaScript.

:Usage:
driver.execute_script(‘document.title’)

file_detector_context(*args, **kwds)
Overrides the current file detector (if necessary) in limited context.
Ensures the original file detector is set afterwards.

Example:

with webdriver.file_detector_context(UselessFileDetector):
someinput.send_keys(‘/etc/hosts’)

:Args:

  • file_detector_class – Class of the desired file detector. If the
    class is different
    from the current file_detector, then the class is instantiated with
    args and kwargs
    and used as a file detector during the duration of the context
    manager.
  • args – Optional arguments that get passed to the file detector class
    during
    instantiation.
  • kwargs – Keyword arguments, passed the same way as args.

find_element(self, by=’id’, value=None)
‘Private’ method used by the find_element_by_* methods.

:Usage:
Use the corresponding find_element_by_* instead of this.

:rtype: WebElement

find_element_by_class_name(self, name)
Finds an element by class name.

:Args:

  • name: The class name of the element to find.

:Usage:
driver.find_element_by_class_name(‘foo’)

find_element_by_css_selector(self, css_selector)
Finds an element by css selector.

:Args:

  • css_selector: The css selector to use when finding elements.

可靠的线上网赌网站,:Usage:
driver.find_element_by_css_selector(‘#foo’)

find_element_by_id(self, id_)
Finds an element by id.

:Args:

  • id\_ – The id of the element to be found.

:Usage:
driver.find_element_by_id(‘foo’)

find_element_by_link_text(self, link_text)
Finds an element by link text.

:Args:

  • link_text: The text of the element to be found.

:Usage:
driver.find_element_by_link_text(‘Sign In’)

find_element_by_name(self, name)
Finds an element by name.

:Args:

  • name: The name of the element to find.

:Usage:
driver.find_element_by_name(‘foo’)

find_element_by_partial_link_text(self, link_text)
Finds an element by a partial match of its link text.

:Args:

  • link_text: The text of the element to partially match on.

:Usage:
driver.find_element_by_partial_link_text(‘Sign’)

find_element_by_tag_name(self, name)
Finds an element by tag name.

:Args:

  • name: The tag name of the element to find.

:Usage:
driver.find_element_by_tag_name(‘foo’)

find_element_by_xpath(self, xpath)
Finds an element by xpath.

:Args:

  • xpath – The xpath locator of the element to find.

:Usage:
driver.find_element_by_xpath(‘//div/td[1]’)

find_elements(self, by=’id’, value=None)
‘Private’ method used by the find_elements_by_* methods.

:Usage:
Use the corresponding find_elements_by_* instead of this.

:rtype: list of WebElement

find_elements_by_class_name(self, name)
Finds elements by class name.

:Args:

  • name: The class name of the elements to find.

:Usage:
driver.find_elements_by_class_name(‘foo’)

find_elements_by_css_selector(self, css_selector)
Finds elements by css selector.

:Args:

  • css_selector: The css selector to use when finding elements.

:Usage:
driver.find_elements_by_css_selector(‘.foo’)

find_elements_by_id(self, id_)
Finds multiple elements by id.

:Args:

  • id\_ – The id of the elements to be found.

:Usage:
driver.find_elements_by_id(‘foo’)

find_elements_by_link_text(self, text)
Finds elements by link text.

:Args:

  • link_text: The text of the elements to be found.

:Usage:
driver.find_elements_by_link_text(‘Sign In’)

find_elements_by_name(self, name)
Finds elements by name.

:Args:

  • name: The name of the elements to find.

:Usage:
driver.find_elements_by_name(‘foo’)

find_elements_by_partial_link_text(self, link_text)
Finds elements by a partial match of their link text.

:Args:

  • link_text: The text of the element to partial match on.

:Usage:
driver.find_element_by_partial_link_text(‘Sign’)

find_elements_by_tag_name(self, name)
Finds elements by tag name.

:Args:

  • name: The tag name the use when finding elements.

:Usage:
driver.find_elements_by_tag_name(‘foo’)

find_elements_by_xpath(self, xpath)
Finds multiple elements by xpath.

:Args:

  • xpath – The xpath locator of the elements to be found.

:Usage:
driver.find_elements_by_xpath(“//div[contains(@class, ‘foo’)]”)

forward(self)
Goes one step forward in the browser history.

:Usage:
driver.forward()

get(self, url)
Loads a web page in the current browser session.

get_cookie(self, name)
Get a single cookie by name. Returns the cookie if found, None if not.

:Usage:
driver.get_cookie(‘my_cookie’)

get_cookies(self)
Returns a set of dictionaries, corresponding to cookies visible in the
current session.

:Usage:
driver.get_cookies()

get_log(self, log_type)
Gets the log for a given log type

:Args:

  • log_type: type of log that which will be returned

:Usage:
driver.get_log(‘browser’)
driver.get_log(‘driver’)
driver.get_log(‘client’)
driver.get_log(‘server’)

get_screenshot_as_base64(self)
Gets the screenshot of the current window as a base64 encoded string
which is useful in embedded images in HTML.

:Usage:
driver.get_screenshot_as_base64()

get_screenshot_as_file(self, filename)
Gets the screenshot of the current window. Returns False if there is
any IOError, else returns True. Use full paths in your filename.

:Args:

  • filename: The full path you wish to save your screenshot to.

:Usage:
driver.get_screenshot_as_file(‘/Screenshots/foo.png’)

get_screenshot_as_png(self)
Gets the screenshot of the current window as a binary data.

:Usage:
driver.get_screenshot_as_png()

get_window_position(self, windowHandle=’current’)
Gets the x,y position of the current window.

:Usage:
driver.get_window_position()

get_window_size(self, windowHandle=’current’)
Gets the width and height of the current window.

:Usage:
driver.get_window_size()

implicitly_wait(self, time_to_wait)
Sets a sticky timeout to implicitly wait for an element to be found,
or a command to complete. This method only needs to be called one
time per session. To set the timeout for calls to
execute_async_script, see set_script_timeout.

:Args:

  • time_to_wait: Amount of time to wait (in seconds)

:Usage:
driver.implicitly_wait(30)

maximize_window(self)
Maximizes the current window that webdriver is using

refresh(self)
Refreshes the current page.

:Usage:
driver.refresh()

save_screenshot = get_screenshot_as_file(self, filename)
Gets the screenshot of the current window. Returns False if there is
any IOError, else returns True. Use full paths in your filename.

:Args:

  • filename: The full path you wish to save your screenshot to.

:Usage:
driver.get_screenshot_as_file(‘/Screenshots/foo.png’)

set_page_load_timeout(self, time_to_wait)
Set the amount of time to wait for a page load to complete
before throwing an error.

:Args:

  • time_to_wait: The amount of time to wait

:Usage:
driver.set_page_load_timeout(30)

set_script_timeout(self, time_to_wait)
Set the amount of time that the script should wait during an
execute_async_script call before throwing an error.

:Args:

  • time_to_wait: The amount of time to wait (in seconds)

:Usage:
driver.set_script_timeout(30)

set_window_position(self, x, y, windowHandle=’current’)
Sets the x,y position of the current window. (window.moveTo)

:Args:

  • x: the x-coordinate in pixels to set the window position
  • y: the y-coordinate in pixels to set the window position

:Usage:
driver.set_window_position(0,0)

set_window_size(self, width, height, windowHandle=’current’)
Sets the width and height of the current window. (window.resizeTo)

:Args:

  • width: the width in pixels to set the window to
  • height: the height in pixels to set the window to

:Usage:
driver.set_window_size(800,600)

start_client(self)
Called before starting a new session. This method may be overridden
to define custom startup behavior.

start_session(self, desired_capabilities, browser_profile=None)
Creates a new session with the desired capabilities.

:Args:

  • browser_name – The name of the browser to request.
  • version – Which browser version to request.
  • platform – Which platform to request the browser on.
  • javascript_enabled – Whether the new session should support
    JavaScript.
  • browser_profile – A
    selenium.webdriver.firefox.firefox_profile.FirefoxProfile object. Only
    used if Firefox is requested.

stop_client(self)
Called after executing a quit command. This method may be overridden
to define custom shutdown behavior.

switch_to_active_element(self)
Deprecated use driver.switch_to.active_element

switch_to_alert(self)
Deprecated use driver.switch_to.alert

switch_to_default_content(self)
Deprecated use driver.switch_to.default_content

switch_to_frame(self, frame_reference)
Deprecated use driver.switch_to.frame

switch_to_window(self, window_name)
Deprecated use driver.switch_to.window

Data descriptors inherited from
selenium.webdriver.remote.webdriver.WebDriver:

__dict__
dictionary for instance variables (if defined)

__weakref__
list of weak references to the object (if defined)

application_cache
Returns a ApplicationCache Object to interact with the browser app cache

current_url
Gets the URL of the current page.

:Usage:
driver.current_url

current_window_handle
Returns the handle of the current window.

:Usage:
driver.current_window_handle

desired_capabilities
returns the drivers current desired capabilities being used

file_detector

log_types
Gets a list of the available log types

:Usage:
driver.log_types

mobile

name
Returns the name of the underlying browser for this instance.

:Usage:

  • driver.name

orientation
Gets the current orientation of the device

:Usage:
orientation = driver.orientation

page_source
Gets the source of the current page.

:Usage:
driver.page_source

switch_to

title
Returns the title of the current page.

:Usage:
driver.title

window_handles
Returns the handles of all windows within the current session.

:Usage:
driver.window_handles