JCrawler Forum Tag: jcrawler

JCrawler Forum Tag: jcrawler http://www.pixelschieber.ch/forum/ Help forums for JCrawler en Fri, 30 Jul 2010 23:11:08 +0000 patrick on "a little help" http://www.pixelschieber.ch/forum/topic/a-little-help#post-437 Mon, 25 Jan 2010 22:38:13 +0000 patrick 437@http://www.pixelschieber.ch/forum/ do you have some SEF components installed? else your mysql server crashed, thy to crawl with less parallel connections patrick james on "a little help" http://www.pixelschieber.ch/forum/topic/a-little-help#post-435 Fri, 22 Jan 2010 14:22:40 +0000 james 435@http://www.pixelschieber.ch/forum/ hello, I have tested this plugin with my joomla version: 1.5.15 I recommend putting in the following fields: levels parallels max connections timeout thank you very much for your help! I drop this error on page after crawling: Warning: Invalid argument supplied for foreach() in /homepages/43/d301128385/htdocs/plugins/system/rereplacer.php on line 939 * httpcode: 500 on url <a href="http://www.masmallorca.es/en/places-of-special-interest.html" rel="nofollow">http://www.masmallorca.es/en/places-of-special-interest.html</a> * httpcode: 500 on url <a href="http://www.masmallorca.es/en/culture.html" rel="nofollow">http://www.masmallorca.es/en/culture.html</a> * httpcode: 500 on url <a href="http://www.masmallorca.es/en/our-people.html" rel="nofollow">http://www.masmallorca.es/en/our-people.html</a> * httpcode: 500 on url <a href="http://www.masmallorca.es/en/gastronomy.html" rel="nofollow">http://www.masmallorca.es/en/gastronomy.html</a> * httpcode: 500 on url <a href="http://www.masmallorca.es/en/local-produce.html" rel="nofollow">http://www.masmallorca.es/en/local-produce.html</a> * httpcode: 500 on url <a href="http://www.masmallorca.es/en/fauna.html" rel="nofollow">http://www.masmallorca.es/en/fauna.html</a> * httpcode: 500 on url <a href="http://www.masmallorca.es/en/flora.html" rel="nofollow">http://www.masmallorca.es/en/flora.html</a> * httpcode: 500 on url <a href="http://www.masmallorca.es/en/excursions.html" rel="nofollow">http://www.masmallorca.es/en/excursions.html</a> * httpcode: 500 on url <a href="http://www.masmallorca.es/en/cycletourism.html" rel="nofollow">http://www.masmallorca.es/en/cycletourism.html</a> * httpcode: 500 on url <a href="http://www.masmallorca.es/ca.html" rel="nofollow">http://www.masmallorca.es/ca.html</a> * httpcode: 500 on url <a href="http://www.masmallorca.es/es.html" rel="nofollow">http://www.masmallorca.es/es.html</a> * httpcode: 500 on url <a href="http://www.masmallorca.es/en.html" rel="nofollow">http://www.masmallorca.es/en.html</a> * httpcode: 500 on url <a href="http://www.masmallorca.es/de.html" rel="nofollow">http://www.masmallorca.es/de.html</a> Etc Etc etc * Error Loading ModulesMySQL server has gone away SQL=SELECT id, title, module, position, content, showtitle, control, params FROM jos_modules AS m LEFT JOIN jos_modules_menu AS mm ON mm.moduleid = m.id WHERE m.published = 1 AND m.access <= 2 AND m.client_id = 1 ORDER BY position, ordering Mensaje * There are 427 links in your sitemap. * Success, wrote /homepages/43/d301128385/htdocs/sitemap.xml * Success, wrote /homepages/43/d301128385/htdocs//administrator/components/com_jcrawler/config.xml * total time: 82.4506 seconds * Success, wrote /homepages/43/d301128385/htdocs/components/com_jcrawler/sitemap.html Bartje on "JCrawler 1.7 does not return a success page" http://www.pixelschieber.ch/forum/topic/jcrawler-17-does-not-return-a-success-page#post-254 Sun, 10 May 2009 10:24:48 +0000 Bartje 254@http://www.pixelschieber.ch/forum/ Would be great.. might be an option to catch the errorlist in a text file instead? Gé patrick on "JCrawler 1.7 does not return a success page" http://www.pixelschieber.ch/forum/topic/jcrawler-17-does-not-return-a-success-page#post-252 Sat, 09 May 2009 19:22:39 +0000 patrick 252@http://www.pixelschieber.ch/forum/ ok, I think there are severals errors and the list of these error\'s is very long. Too long for IE. I will do an option to hide these error\'s its not a big thing. Greets Patrick Bartje on "JCrawler 1.7 does not return a success page" http://www.pixelschieber.ch/forum/topic/jcrawler-17-does-not-return-a-success-page#post-250 Thu, 07 May 2009 17:54:01 +0000 Bartje 250@http://www.pixelschieber.ch/forum/ Hi patrick, 1 level works fine, creation of 19 links in the sitemap. Jcrawler returns the result page with the usual information.. 3 levels generates 903 links, but no return to the results as you already know. I\'m looking forward to the next built or release :) by the way, I run IE8 (same problem as IE7?) Gé patrick on "JCrawler 1.7 does not return a success page" http://www.pixelschieber.ch/forum/topic/jcrawler-17-does-not-return-a-success-page#post-246 Wed, 06 May 2009 21:06:34 +0000 patrick 246@http://www.pixelschieber.ch/forum/ Hey G Driever, Can you try to generate a sitemap just for the first level? Maxconnections between 50 and 100 Sometimes IE7 has a problems with lage pages (maybe you recieve a large error list or something like that), firefox? Technical there is no problem to schedule the creation, but there is a heavy server load. The next feature will be a html output! :) Greets Patrick Bartje on "JCrawler 1.7 does not return a success page" http://www.pixelschieber.ch/forum/topic/jcrawler-17-does-not-return-a-success-page#post-242 Tue, 05 May 2009 16:47:06 +0000 Bartje 242@http://www.pixelschieber.ch/forum/ .., I really love the tool, and the xml map is extensive and accepted by google.. It would be so nice therefore to get a resultpage back after the the sitemap is created. Any suggestion, i tried for days? Btw.. is it, or will it be possible to schedule the creation job on the server? G Driever Bartje on "JCrawler 1.7 does not return a success page" http://www.pixelschieber.ch/forum/topic/jcrawler-17-does-not-return-a-success-page#post-238 Thu, 30 Apr 2009 21:49:53 +0000 Bartje 238@http://www.pixelschieber.ch/forum/ ?? Whatever I try, i don\'t get it fixed. Is there any pre-condition/configuration (php, mysql, joomla, IIS, ..) that i have missed? by the way, the Jcrawler page shows the message \"..waiting for <a href="http://../administrator/index2.php.." rel="nofollow">http://../administrator/index2.php..</a>\" Bartje on "JCrawler 1.7 does not return a success page" http://www.pixelschieber.ch/forum/topic/jcrawler-17-does-not-return-a-success-page#post-225 Mon, 27 Apr 2009 15:05:07 +0000 Bartje 225@http://www.pixelschieber.ch/forum/ Patrick, Some more information: I have tried to generate sitemaps with very short timeouts (1 till 15 sec\'s), and indeed, after the sitemap is created, the \'success page\' shows up, with some errors, and a very limited output in de sitemap.xml file. When increasing the timeout, the sitemap grows to a complete url-list, but then the progress gif (crawling ...) won\'t disappear. Finally the browser quits (\"internet explorer cannot display the webpage\") I checked the php ini file on my server (no strange things, curl ok, fopen ok) and have experimented with some maximum settings, but with no results. If you like i can povide you of the php-log file during the latest attempt (50 parallel connections, 50 sec timeout) Greetings, Gé Driever dangelo211 on "JCrawler 1.7 does not return a success page" http://www.pixelschieber.ch/forum/topic/jcrawler-17-does-not-return-a-success-page#post-222 Mon, 27 Apr 2009 03:06:26 +0000 dangelo211 222@http://www.pixelschieber.ch/forum/ Please help in installing Jcrawler. I don\'t know what to do after I downloaded the application. Bartje on "JCrawler 1.7 does not return a success page" http://www.pixelschieber.ch/forum/topic/jcrawler-17-does-not-return-a-success-page#post-221 Sun, 26 Apr 2009 21:11:13 +0000 Bartje 221@http://www.pixelschieber.ch/forum/ When starting the creation proces from the backend admin there is a sitemap created, but the the proces never stops and the screen doen\'t change, meaning, the screen doesn\'t turn into the success page (from where the sitemap.xml might be sent to the 5 search engines). Though, very incidently, the screen is shown, in particular (i guess) when there are a lot of errors on links (i.e. timeouts) and the xml-file is limited. (Otherwise i wouldn\'t be aware of the success page at all :) Server settings: Win SBS 2003, PHP 5.2.8, Joomla 1.5.9, JCrawler 1.7 beta patrick on "JCrawler 1.7 does not return a success page" http://www.pixelschieber.ch/forum/topic/jcrawler-17-does-not-return-a-success-page#post-219 Sun, 26 Apr 2009 16:56:31 +0000 patrick 219@http://www.pixelschieber.ch/forum/ How do you mean that \"here is no return to the success page\". Can you explain to me what you see when JCrawler is done? Greets Patrick Bartje on "JCrawler 1.7 does not return a success page" http://www.pixelschieber.ch/forum/topic/jcrawler-17-does-not-return-a-success-page#post-215 Thu, 23 Apr 2009 21:07:16 +0000 Bartje 215@http://www.pixelschieber.ch/forum/ I installed JCrawler 1.7 (because of the PHP 5.x compatibility). When starting generating a sitemap all seem to work well, i see a sitemap being generated (which looks pretty complete), but unfortunately there is no return to the success page. Any suggestions? darkside1it on "Jcrawler doesn\\\'t generate a sitemap" http://www.pixelschieber.ch/forum/topic/jcrawler-doesnt-generate-a-sitemap#post-179 Wed, 01 Apr 2009 11:55:21 +0000 darkside1it 179@http://www.pixelschieber.ch/forum/ hi guys...i solved...i used this: <code><a href="http://www.xml-sitemaps.com/" rel="nofollow">http://www.xml-sitemaps.com/</a> </code> Bye darkside1it on "Jcrawler doesn\\\'t generate a sitemap" http://www.pixelschieber.ch/forum/topic/jcrawler-doesnt-generate-a-sitemap#post-176 Mon, 30 Mar 2009 19:33:42 +0000 darkside1it 176@http://www.pixelschieber.ch/forum/ hi rogier. i try it but it does not work .... however thank you Nobody can help me?? Thank you Daniele Rogier on "Jcrawler doesn\\\'t generate a sitemap" http://www.pixelschieber.ch/forum/topic/jcrawler-doesnt-generate-a-sitemap#post-175 Mon, 30 Mar 2009 17:20:35 +0000 Rogier 175@http://www.pixelschieber.ch/forum/ Take a look here: <a href="http://www.pixelschieber.ch/forum/topic/installed-component-pressed-start-got-a-404" rel="nofollow">http://www.pixelschieber.ch/forum/topic/installed-component-pressed-start-got-a-404</a> Rogier on "Installed Component, pressed START got a 404" http://www.pixelschieber.ch/forum/topic/installed-component-pressed-start-got-a-404#post-174 Mon, 30 Mar 2009 16:10:45 +0000 Rogier 174@http://www.pixelschieber.ch/forum/ Try to chmod 666 robots.txt and sitemap.xml If that does not work, try to enable Search Engine Friendly urls and Use Apache mod_rewrite. MENU: Global Configuration -> Site If this does not work... I don\'t know... mcavani on "Installed Component, pressed START got a 404" http://www.pixelschieber.ch/forum/topic/installed-component-pressed-start-got-a-404#post-173 Mon, 30 Mar 2009 15:52:24 +0000 mcavani 173@http://www.pixelschieber.ch/forum/ Hello JCrawler Crew, Newbie here so please bare with me. I\'m not sure what I\'m doing wrong here. I went into my server and created the /cg-la.com/sitemap.xml but I still get the 404 (Page not found message) when asking the component to crawl my site. thanks! darkside1it on "Jcrawler doesn\\\'t generate a sitemap" http://www.pixelschieber.ch/forum/topic/jcrawler-doesnt-generate-a-sitemap#post-167 Mon, 30 Mar 2009 10:35:40 +0000 darkside1it 167@http://www.pixelschieber.ch/forum/ hi Patrick. Have you find something? Thank you so much Daniele darkside1it on "Jcrawler doesn\\\'t generate a sitemap" http://www.pixelschieber.ch/forum/topic/jcrawler-doesnt-generate-a-sitemap#post-165 Sat, 28 Mar 2009 10:46:24 +0000 darkside1it 165@http://www.pixelschieber.ch/forum/ here it is: <a href="http://www.culturaurbana.it" rel="nofollow">http://www.culturaurbana.it</a> Thank you patrick on "Jcrawler doesn\\\'t generate a sitemap" http://www.pixelschieber.ch/forum/topic/jcrawler-doesnt-generate-a-sitemap#post-162 Fri, 27 Mar 2009 20:13:48 +0000 patrick 162@http://www.pixelschieber.ch/forum/ hey Daniele, can you give me your url? so I can hava a look. I think its a known problem. greets Patrick darkside1it on "Jcrawler doesn\\\'t generate a sitemap" http://www.pixelschieber.ch/forum/topic/jcrawler-doesnt-generate-a-sitemap#post-161 Fri, 27 Mar 2009 00:37:40 +0000 darkside1it 161@http://www.pixelschieber.ch/forum/ hi patrick. Thanks for your help. When i go to the page(obviously the url is not mysite.it ;-)) isee: <pre><code> Not Found The requested URL /httpdocs/sitemap.xml was not found on this server. </code></pre> This give me the idea that i wrote the wrong document root, and so it was...i wrote the wrong document root ;-) I have changed it and now after i generate a sitemape with the component it tells me: <pre><code> <code> * There are 0 links in your sitemap. * Success, wrote /var/www/vhosts/mysite.it/httpdocs/sitemap.xml * Success, wrote /var/www/vhosts/mysite.it/httpdocs//administrator/components/com_jcrawler/config.xml * total time: 0.6339 seconds</code> </code></pre> Now it referres to the correct path and files and give me no errors, but as you can see i don\\\'t have link in the sitemap. uhm...what do u think? Thank you Daniele patrick on "Jcrawler doesn\\\'t generate a sitemap" http://www.pixelschieber.ch/forum/topic/jcrawler-doesnt-generate-a-sitemap#post-160 Thu, 26 Mar 2009 23:24:32 +0000 patrick 160@http://www.pixelschieber.ch/forum/ Ciao Daniele, can you go on your sitemap <a href="http://www.mysite.it/httpdocs/sitemap.xml" rel="nofollow">http://www.mysite.it/httpdocs/sitemap.xml</a> and then view the sourcecode? What do you see? Are there any links? greets Patrick darkside1it on "Jcrawler doesn\\\'t generate a sitemap" http://www.pixelschieber.ch/forum/topic/jcrawler-doesnt-generate-a-sitemap#post-157 Thu, 26 Mar 2009 19:35:29 +0000 darkside1it 157@http://www.pixelschieber.ch/forum/ ok. I did not know i must have all the files(robots.txt,sitemap.xml,config.xml)with 777 permission. I changed the permission but now Jcrawler creates a sitemap with only an url... When i click on:View my sitemap I see a white page with on the top this words: <code><a href="http://www.mysite.it2009-03-26T01:28:02Z" rel="nofollow">http://www.mysite.it2009-03-26T01:28:02Z</a> 0.5 daily</code> Ah... just to know you ,i am using a SEF component(ARTIO Joomsef) Thank you Guys Regards Daniele darkside1it on "Jcrawler doesn\\\'t generate a sitemap" http://www.pixelschieber.ch/forum/topic/jcrawler-doesnt-generate-a-sitemap#post-156 Thu, 26 Mar 2009 13:34:09 +0000 darkside1it 156@http://www.pixelschieber.ch/forum/ First of all. Hi to all :-) I have a problem with JCrawler, after i start to generate my sitemap i receive this message: <ul> <li>Before i see this 4 errors</li> </ul> <pre><code> * httpcode: 404 on url <a href="http://mysite.it/skype%3Awww.mysite.it?&lt;br" rel="nofollow">http://mysite.it/skype%3Awww.mysite.it?&lt;br</a> /> * httpcode: 404 on url <a href="http://mysite.it/skype%3Awww.mysite.it?&lt;br" rel="nofollow">http://mysite.it/skype%3Awww.mysite.it?&lt;br</a> /> * httpcode: 404 on url <a href="http://www.mysite.it/components/com_virtuemart/shop_image/product/o&lt;br" rel="nofollow">http://www.mysite.it/components/com_virtuemart/shop_image/product/o&lt;br</a> /> * httpcode: 404 on url <a href="http://www.mysite.it/components/com_virtuemart/shop_image/product/t-shirt%20o&lt;br" rel="nofollow">http://www.mysite.it/components/com_virtuemart/shop_image/product/t-shirt%20o&lt;br</a> /></code></pre> <ul> <li>Down of that there is this message of accomplished generation</li> </ul> <pre><code> * There are 128 links in your sitemap. * Success, wrote <a href="http://www.mysite.it/httpdocs/sitemap.xml&lt;br" rel="nofollow">http://www.mysite.it/httpdocs/sitemap.xml&lt;br</a> /> * Success, wrote <a href="http://www.mysite.it/httpdocs//administrator/components/com_jcrawler/config.xml&lt;br" rel="nofollow">http://www.mysite.it/httpdocs//administrator/components/com_jcrawler/config.xml&lt;br</a> /> * total time: 84.6516 seconds * robots.txt contains already sitemap location</code></pre> I think that the first 4 errors are not relevant. The problem is that i can\\\'t see any sitemap when i check it . Why? Thanks to all Regards Daniele patrick on "FAQ - Please read it before you post" http://www.pixelschieber.ch/forum/topic/faq-please-read-it-before-you-post#post-3 Sat, 27 Dec 2008 21:49:03 +0000 patrick 3@http://www.pixelschieber.ch/forum/ Please read the FAQ before you post. When you do a post, please post your <ul> <li>url</li> <li>configuration</li> <li>problem exactly</li> </ul> thank you! Maybe you will find the solution. 1. I get HTTP/1.1 403 FORBIDDEN error on all links? Why is this? Try to uncomment the “RewriteBase /” line in your .htaccess file 2. I get httpcode: 404 on url …. make sure this url is availabe to the crawler errors There are invalid links on your site. 3. When i generate the xml it returns a file with no urls inside, and the message: There are 0 links in your sitemap. Success, wrote /home/www/domain.org/sitemap.xml Success, wrote /home/www/domain.org//administrator/components/com_jcrawler/config.xml Be ensure, that your site is not in the offline mode! 4. How does the internal ranking work? Read this: <a href="http://www.ianrogers.net/google-page-rank/" rel="nofollow">http://www.ianrogers.net/google-page-rank/</a> FAQ will be extended soon.