Jump to content
[MUST READ] Forum Rules ×

download content with request and restore robots.txt


Rogelio

Recommended Posts

Hi, I accidentally delete my file robots.txt, i want restore, I will restore on /htdocs/robots.txt or /robots.txt??

I have small web app  and i need download content with api as python requests, but when i download the content always get:

<html><body><script type="text/javascript" src="/aes.js" ></script><script>function toNumbers(d){var e=[];d.replace(/(..)/g,function(d){e.push(parseInt(d,16))});return e}function toHex(){for(var d=[],d=1==arguments.length&&arguments[0].constructor==Array?arguments[0]:arguments,e="",f=0;f<d.length;f++)e+=(16>d[f]?"0":"")+d[f].toString(16);return e.toLowerCase()}var a=toNumbers("adsadasdasdasda"),b=toNumbers("dadadsdsadsadasdasd"),c=toNumbers("asdasdadsadaddd");document.cookie="__test="+toHex(slowAES.decrypt(c,2,a,b))+"; expires=Thu, 31-Dec-37 23:55:55 GMT; path=/"; location.href="https://myurl.byethost12.com/suburl?i=1";</script><noscript>This site requires Javascript to work, please enable Javascript in your browser or use a browser with Javascript support</noscript></body></html>

Is it posible download content with apis from shell or script without web browser??

ByetHost it is the best!!

Link to comment
Share on other sites

  • 3 weeks later...

Some messages are set to state about javascript, in case some people have it disabled.
9 times out of 10, the person wont have it disabled, but the message is just in case.
But, like stated, API points can not be created on free service as these requests will be seen as bot requests and blocked, this is however supported on premium plans.
Your Python Code is falling into this, which is why it can't be done, and you are getting the message.

Link to comment
Share on other sites

But I am not trying to create API point (I admit I do not know what you mean by this), just to download the page. So I gather some ways of downloading the page are not allowed, whereas some (like some web browsers) are. I would like to know what way of downloading the page is allowed. If it is not specific web browsers, what is it?

Link to comment
Share on other sites

I tried the following Python code:

urlopen('https://bagr.byethost6.com').read()

By the way, I do not know Python, this is just a line in a tutorial, so I do not know exactly what urlopen does. It returned the response from OP's post.

I also tried plain curl:

$ curl https://bagr.byethost6.com

which gave me a 403 http response.

Elsewhere on this forum there was something about javascript challenge on request, which would explain the behavior, but I cannot find it officially mentioned and documented anywhere.

There are no scripts on the page itself, it is just plain HTML.

Link to comment
Share on other sites

Guest
This topic is now closed to further replies.
×
×
  • Create New...