How to use the Development Portal

Accessing the website #

To log on to the Development Portal, you must go to the following address:

• Development Portal (appserv.protoweb.org)

You will need a user account set up before you can log in. Please see this article for more information.

Troubleshooting #

The development portal has been tested and is supported in the following web browsers:

• Mozilla Firefox
• Google Chrome
• Microsoft Edge

If you receive SSL certificate issues, please do the following:

• Make sure you are using a supported web browser.
• Close the tab to the page.
• Clear all cache on the web browser.
• Open the web site again, making sure that you specifically type in https:// as the prefix to the address.

Main Menu #

Archive web site #

Start a new restoration project here. This button will allow you to start a new scraping process from The Internet Archive. You can specify which URL to download, select a target date for the files, and how many levels deep to recurse a website.

View web sites #

View all your current restoration projects here. Here you can view ongoing scraping processes, edit project files, and publish projects when they are ready. You may also access and fix other websites on Protoweb, but currently this access is only given by administrators on a case-by-case basis.

View FTP sites #

Edit existing FTP sites with the File Manager. An Admin can create new FTP sites from scratch, and access can be granted as needed.

How to archive a web site #

From the main menu, if you click on the option Archive web site, you will be brought to a form that allows you to start a new website rebuilding process. If you have previously used the Warnick Mirroring Tool, this form contains the typical parameters you would normally give as parameters.

Domain Name to Save #

Type in the website domain name here. The domain name must be given in the following form without http or path structure: www.domain.com, or www.somesite.org, etc. Do not write the protocol with the address (such as http://www.domain.com/). Also, do not type the folder structure, eg. www.domain.com/path/to/files/ to archive a specific directory. You can only archive a whole domain. An admin can help archiving a single folder or a folder substructure, if it is needed. This limitation in the development portal may change in the future.

Targeted Date #

Specify the historic date for the website which the Warnick Mirroring Tool will try to use when looking for pages to download from The Internet Archive’s Wayback Machine. Your project will have a better success rate, if you go to the Internet Archive’s Wayback Machine and use it to research the best available copy of a website by moving through their timeline and choosing a date where the website appears most complete. Once you have selected a good date for the website, type the date in this field.

Link Traversal Depth #

Warnick counts how many links it will traverse starting from the homepage. This tells Warnick to start from the main page, and then go through n number of pages.

How link traversal works #

If you choose the Link Traversal Depth to be depth=2, it will tell the Warnick website scraper to download pages only two links deep. Warnick will do the following. It will start from the homepage, then download it (home page is always depth=1). It will then click on the first link on the homepage and download that page. The downloaded page will now be at depth=2. Then, if that page has another link, the page will not be downloaded, because it would be at depth=3, which is more than what is allowed by Traversal Depth. The ignored page may still be downloaded however, if it is directly accessible through the home page, at which point it would be seen as depth=2.

For most sites, Link Traversal Depth 10 is fine. Use a larger number if you know there are some pages in the website that take more than 10 clicks to reach starting from the homepage. Use a smaller number, if you are sure that the website needs less.

Start Archiver #

Start page archival process. If everything is filled in OK, you will see the job show up in the web site manager.

Web site manager #

When you click on the View web sites from the main menu, you will be brought to the web site manager. Here you can view ongoing Warnick website scraper archival jobs. You can also choose the web site you would like to edit.

Active Jobs #

This section (not shown in the screenshot) shows a list of ongoing Website scraping processes. Whenever you have an ongoing Warnick web site scraping process, it will show up in this section at the top of the list. You can view the log to monitor the process, or cancel the job, if you are the owner.

Completed Jobs & Web sites in development #

This section (shown in the screenshot) shows all web sites that have completed the scraping process successfully. You may browse these web sites directly on the development server. These web sites are not accessible to the public on the production servers.

Each site on the list has action buttons that you can use to manage a web site.

• View Log – You can View the log of a completed mirror process.

In addition, site owners have additional functions at their disposal:

• Edit – If you are the owner, you may Edit the web site at this time. This will open a File Manager, that allows you to edit the files on the web site. The File Manager has a code-highlighting Advanced Editor, that allows you to easily edit HTML code. The File Manager also has other powerful functionalities, such as the ability to Zip or UnZip individual files or a whole site. You can also upload additional files, or upload from URL, if the file is available on the Internet.
• Publish – When you have finished restoring a website, you may click on Publish to make the website available to the public on the production servers. The page will be moved to the Published Sites section. You will continue to retain full access rights to a published web site, except a published site cannot be deleted by a non-admin, so make sure it is complete before you publish it. A published web site is manually added to the inode portal, so please let an Admin know when you have published a site, so that they can add it there to be featured. In many cases published web sites will be featured on the Protoweb website’s showcase, and/or social channels. This is to notify users that a new web site is available, and to credit the contributor who rebuilt the web site.
• Delete – If you need to cancel the project or you notice the archival has failed, and you need to try again, you may click the Delete button. This will remove the web site to allow you to try archiving a page again.

Published Sites #

This section at the bottom of the list (not shown on the screenshot) shows all the websites that have been published with the Development Portal. These websites may be browsed directly on the production server, and they are publicly available on Protoweb.

Each site on the list has action buttons that you can use to manage a web site.

• View Log – You can View the log of a published web site. This is only useful for debugging purposes.

In addition, site owners have additional functions at their disposal:

• Edit – If you are the owner, you may Edit the web site. See above for a detailed explanation of the Edit button.

Note! A published web site cannot be deleted by a non-admin, so make sure it is complete before you publish it.

FTP site manager #

When you click on the View FTP sites from the main menu, you will be brought to the FTP site manager. Here you can choose an FTP sites that you would like to edit.

FTP Site List #

This section lists all the FTP sites that have been configured with the FTP site manager. If you need access to a specific FTP site, please contact an Admin.

If you are marked the owner of a specific FTP site, you will have an action button that will let you manage an FTP site.

• Manage – If you are the owner, you may Manage the FTP site. This will open a File Manager, that allows you to download, upload and edit the files on the site. The File Manager has a code-highlighting Advanced Editor, that allows you to easily edit HTML code. The File Manager also has other powerful functionalities, such as the ability to Zip or UnZip individual files or a whole site. You can also upload additional files, or upload from URL, if the file is available on the Internet.

Special functionalities in FTP sites #

Some FTP sites have a greeting message when the directory is first opened. Visit ftp.netscape.com or ftp.3drealms.com on Protoweb for an example of this.

• Greeting – To add a greeting message, such as the one on top, open the File Manager to the FTP site you would like to Manage, and create a new empty file called README all uppercase. Edit this file, and you may then add a plaintext greeting message that will show up when an FTP site is opened.
• Header – If you would like to add an additional message before the file listing, create a file HEADER all uppercase and add your header message here in plaintext.
• Footer – If you would like to add an additional message after the file listing, create a file FOOTER all uppercase and add your header message here in plaintext.