повторить 11 страниц
источник: https_www_wmchealth_org_physical_locator
шаги:< /p>
- имя: fetch_initial_asp_values
type: capture_values_request
results_type: html
refresh_interval:
type: Every_queue_loop_iterations # один раз, Every_queue_loop_iteration или продолжительность (15 минут, 30 минут) — по умолчанию один раз
значение: 1 # значение, которое применяется к типу (см. раздел интервала обновления ниже)
url:
headers: _use_key :headers
uri: https://www.wmchealth.org/physical-locator
results_key: capture_first_step
results_transformer: ${transform_first_step(captured_first_step)
< li>name: fetch_ Second_form_asp_values
type: capture_values_request
results_type: html
refresh_interval:
type: Every_queue_loop_iterations # один раз, каждая_очередь_цикла_итерация или продолжительность (15 минут, 30 минут) — по умолчанию — один раз
значение: 1
url:
метод: POST
post_body: FORM
headers: _use_key:headers
uri: https:// www.wmchealth.org/physical-locator
params:
__EVENTTARGET: cpsys_FormItem_PhysicianSearchLeft_SearchBtn
__EVENTARGUMENT:
__VIEWSTATE: ${captured_first_step.results.viewState}
__VIEWSTATEGENERATOR: ${ capture_first_step.results.viewGen__VIEWSTATEENCRYPTED:
__EVENTVALIDATION: ${captured_first_step.results.eventValidation}
HtmlSearchCriteria:
cpsys_FormItem_PhysicianSearchLeft_SpecialtySearch:
cpsys_FormItem_PhysicianSearchLeft _NameSearch:
cpsys_FormItem_PhysicianSearchLeft_KeywordSearch:
cpsys_FormItem_PhysicianSearchLeft_ZipCode:
cpsys_FormItem_PhysicianSearchLeft_LanguageSearch:
cpsys_FormItem_PhysicianSearchLeft_GenderSearch:
cpsys_FormItem_PhysicianSearchLeft_Distance: 30
ctl00$ ctl00$FormAction: ExecuteSearch
ctl00$ctl00$FormGroup: PhysicianSearchLeft
ctl00$ctl00$FormButton: SearchBtn
results_key: capture_ Second_step
results_transformer: ${transform_ Second_step(captured_ Second_step) - name: fetch list
type: html_request
Разбиение на страницы:
тип: auto
max_total_rows_overall : 110
results_path_to_total_results: ${captured_ Second_step.results.count} # приоритет над results_path_to_total_pages
url:
метод: POST
post_body: FORM
headers: _use_key:headers
uri: https://www.wmchealth.org/physical-locator
params: ${get_params( capture_ Second_step, pagination.current)results_key: поставщики
results_path_to_rows: ${xpath_select(".//div[@class='info']/h2", поставщики)row_processors:
имя: Запрос страницы профиля
тип: дополнительный_html_запрос
url:
headers: _use_key:headersuri: https://www.wmchealth.org/${xpath_attribute_get_text(».//a», 'href', row)}
results_key: Profile_page - имя: поля схемы данных JSON карты
тип: Schema_mapper
results_key: row_transformed
карта:
источник: ${source}
site_type: H
full_name: ${xpath_get_text("//div[@class='page-content mod-details dir']//h1", Profile_page) - name: сохранить в csv
тип: csv_writer
ключ: row_transformed
заголовки:
полномочие: www.wmchealth.org/ Physician-locator
принять: text/html,application/xhtml+ xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7
происхождение: https://www.wmchealth.org/physical-locator
пользовательский агент: Mozilla/5.0 (Windows NT 10.0; Вин64; x64) AppleWebKit/537.36 (KHTML, например Gecko) Chrome/121.0.0.0 Safari/537.36
custom_expressions: |
def Transform_first_step(results):
htmlObj = results.results
viewState = xpath_attribute_get_text("//input[@id='__VIEWSTATE']", 'value', htmlObj)
viewGen = xpath_attribute_get_text("//input[@id='__VIEWSTATEGENERATOR']", 'value', htmlObj )
eventValidation = xpath_attribute_get_text("//input[@id='__EVENTVALIDATION']", 'value', htmlObj)
dic = {
"viewState": viewState,
"viewGen": viewGen,
"eventValidation": eventValidation
}
return dic
def Transform_ Second_step(results):
htmlObj = results.results
viewState = xpath_attribute_get_text("//input[@id='__VIEWSTATE']", 'value', htmlObj)
viewGen = xpath_attribute_get_text("//input[@id='__VIEWSTATEGENERATOR']", 'value', htmlObj )
eventValidation = xpath_attribute_get_text("//input[@id='__EVENTVALIDATION']", 'value', htmlObj)
count = Split_to_value(-1, xpath_get_text("//span[@id=) 'cphBody_cphCenter_ctl01_PaginationTop']/div", htmlObj), ' ')
dic = {
"viewState": viewState,
"viewGen": viewGen,
"eventValidation": eventValidation ,
"count": count,
return dic
def get_params(captured_ Second_step, page):
из ячейки математического импорта
if(page = = 1):
dic = {
"__EVENTTARGET": "cpsys_FormItem_PhysDirectory_searchBtn",
"__VIEWSTATE": "{}".format(captured_ Second_step.results.viewState),
" __VIEWSTATEGENERATOR": "{}".format(captured_ Second_step.results.viewGen),
"__EVENTVALIDATION": "{}".format(captured_ Second_step.results.eventValidation),
"ctl00$ctl00$FormAction" : "ExecuteSearch"
}
else:
dic = {
"__EVENTTARGET": "ctl00$ctl00$cphBody$cphCenter$ctl01$PaginationTop$PaginationTop_lb{} ".format(page),
"__VIEWSTATE": "{}".format(captured_ Second_step.results.viewState),
"__VIEWSTATEGENERATOR": "{}".format(captured_ Second_step.results.viewGen) ,
"__EVENTVALIDATION": "{}".format(captured_ Second_step.results.eventValidation),
"ctl00$ctl00$FormAction": "ExecuteSearch"
}
return dic< /li>
Подробнее здесь: https://stackoverflow.com/questions/782 ... lb1-how-to
