Miles Mathis - Fun House Mirror

I was in a weird mood yesterday and couldn't get into any work on my site (I've been working on support pages) and didn't want to play any games or watch a movie. I just didn't really want to do anything. But I can't handle not doing anything so I decided to see if I could scrape the information from Miles' main page and retrieve all of the links to the papers and their descriptions, etc.

It was not easy! I think Miles uses some sort of Content Management System (CMS) that creates horrific HTML. Not the worst I have seen, but still not very well structured. It looks fine to the user, but the code itself is just all over the place. No clear groupings that I could take advantage of, inconsistent sections, font tags spread everywhere and not in clean ways but intermingled with other tags., HTML is really a horrible language and browsers allow some disgusting code to actually work.

In spite of all that, I did actually create an application that can pull out each Section and Paper reference and store it in a database. I then created a few PHP pages that used the database to populate the HTML code. Both of the pages I created contain a menu on the side that contains each Section: Relativity, Gravity, QED and QCD, etc. The pages differed in how they presented the actual paper references. The first just put them all on the page, one after the other but grouped by section. The other page only presented the current section and clicking on the menu would change the list of papers.

I got the first page working such that when you scrolled down the page, the menu would highlight the current section you are looking at. That took a bit of figuring out but was simple enough in the end. I just added an event listener to the scroll event (so it gets called whenever the page is scrolled) and then check the sections to figure out which one is on screen at the moment and update the menu.

I call it a Fun House Mirror because although it has the same content as Miles site, it presents it in a different way and looks quite different. Unfortunately, I can't give you a link to it at the moment as I am still working on it and I will not update my site with it until I have permission from Miles to do so. Even though his site does explicitly say that the content can be reproduced for non-commercial purposes, I want explicit permission since I am scraping his site for the content.

I plan on adding a tagging system to it so that it will actually look at the papers themselves and find references to a preset list of tags and record those links. This will allow a searching mechanism to be used and also a glossary page to be created. Two things that I think are absolutely necessary for working with Miles content.

I am also thinking about an email notification mechanism when new papers are added. This will probably only be for registered users of my site, when I have all of that working. It could also present a notification when you access my site, much like the PM notifications on this forum.

I'm not sure why I am creating even more work for myself, I have enough to do as it is, but sometimes an idea just needs to be dealt with or it will get lost. So I would appreciate any help you guys could give me with this. The first thing I need is a list of tags to search the papers for. At the moment I am just recording the tag name itself but in order for it to be used for other purposes, it is probably a good time to define a more comprehensive structure. I think I only need 2 other fields to support a glossary: definition and source. The definition is just a string that defines the tag. The source is a URL to the online definition and may be omitted. So if you guys can come up with terms to search for, it would help me immensely.

Format: I will use a JSON format for this, as it requires a little bit of structure.

"planet": {
 "definition": "a planet is a celestial body which:"
"caveats": [
   "is in orbit around the Sun,"
   ,"has sufficient mass to assume hydrostatic equilibrium (a nearly round shape), and"
   ,"has \"cleared the neighborhood\" around its orbit."
 ]
 ,"source": "https://en.wikipedia.org/wiki/IAU_definition_of_planet"
 ,"update": false
}

You can have as many caveats in the definition as required or none, whatever the case may be. Some tags will need Mathis definitions, rather than mainstream ones, if he has redefined the term. The source should be the paper that it is defined in but if you can't find it, or can't be bothered to look (which is a huge task sometimes), then just put in "http://www.milesmathis.com". I don't want this to be too onerous on you and we can always fill things in later.

I have added in the "update" property which is a boolean value (true or false), and is optional (defaults to false), to indicate that this tag already exists and needs to be changed, rather than inserted as a new record.

So if you feel like helping out, then just post any you can think of here, in the above format, and I will compile and load them into my database. I will make a post when I have loaded the tags you guys have defined so that you know where I am up to and I will also post the ones that I have defined.

I think this is a really good project that will help many of Miles' readers, especially when Miles has redefined a term. I can focus on parsing the papers to search for the terms and getting some webpages up to view them.

I received an email back from Miles allowing me to publish a mirror of his site. I was just on my way out the door to head to work when I got it though so I will publish it later today or tomorrow night.

I got the keyword search working last night and created a quick page to view the results. It presents the tags on the left side of the page and the rest of the area presents the links to papers that contain that tag. I just copied one of the existing pages I had created for the main page and tailored it to the keyword database. I added in about 40 tags, just titles, no definitions, to test with and everything seemed to work well. The tags are only matched against whole words, so I put in the word 'miles' to see how many matches it found and it found all of the word matches but did not match his email address which is exactly what I wanted. This means that it will not match the word 'time' with words like 'sometimes' which would just bloat the results and be more confusing than helpful. I'm sure there are cases where you do want it to match such things, but it is a small price to pay in my opinion. If I do find one that is absolutely required, then I will add the regular expression pattern to the database for each tag rather than being specified in the code for all tags. Or maybe a mix of both where it can be specified in the DB but if not, then use the standard expression.

This is not exactly the same as a search like google where you can type in anything you want, but it is a start. I was thinking about writing another search algorithm that counted every single word in a paper (except common words like and, or, I, at, etc) and then figure out some sort of ranking system of the words based on their frequency. This is basically the opposite of what I have done now. Currently, it uses a preset list of words to search for but this other one would use a preset list of words to ignore and maybe a preset list of special words that get an extra ranking (such as gravity, charge, spin, etc).

I had a quick look into using a more formal search system (Apache Solr or Apache Lucene - Solr uses Lucene under the hood) but it seems like it would be overkill.

I was also thinking about databatizing all of Miles' papers which would make them more searchable and I could do things like include a snippet of the paragraph containing the search term in the results, much like google. This would also allow me to serve the papers myself and therefore be able to link to a specific section of a page whereas I have no way to do that with Miles' pages since there is no identifier in the code to link to. I'd just have to figure out how to deal with formatting and image placement, etc, if I want to serve the papers myself through a database.

I have published the first cut of my Mathis Mirror which contains 2 views of his main science page and a keywords page.

I noticed that the pages did not work well in my phone browser. The menus did not show and could not be accessed even through the button it provided to do so. I'll look into this problem at some point.

http://www.nevyns-lab.com/mathis

Hi Nevyn.
You can compare your MM index with mine at https://milesmathis.forumotion.com/t21-mm-s-titles-index-by-lloyd. Mine only indexed the Titles. Airman posted that for me last year. I see yours is in numerical order, whereas mine is in alphabetical order. But mine has the page numbers too, so they could be resorted to go in numerical order, if needed. But alphabetical order is good for acting like a glossary etc.

I told Airman how to do a web search of MM's site at https://milesmathis.forumotion.com/t140-collaborate-to-promote-mathis-ideas-better#1009.

Graphic Request
Nevyn, I have some graphics requests for you or anyone who does graphics. I'm doing an MM miniconference today & will probably do more later. And it would help a lot if there were diagrams of a lot of MM's ideas. Actually, video clips would be better in some cases. Like showing how photons and antiphotons enter atomic polar axes and produce electrical conduction or magnetic effects, including how photon and antiphoton spins add or cancel and how that affects magnetism.

Thanks Lloyd, that list will help me flesh out my keywords a bit. I saw your method of using google to search Miles site and I must say that I didn't know that. Kind of makes what I am doing a bit superfluous, but I'll see where it leads anyway. Sometimes it is nice to have a list to select from rather than trying to figure out what to search for.

With respect to graphics, I can't help you at the moment, certainly not for today's conference and not in the near future as I have too much work to do as it is. However, they are great ideas to demonstrate and when the smoke clears from my current work (probably early 2016), I will look into building something to do so. I want a suite of little apps like these to show the concepts in an easy to digest fashion.

While it is not exactly targeted to your current needs, you may be able to use my existing apps to demonstrate at least some of it. You may be able to create a video of you using my apps that leads the viewer through whatever you are trying to show. It will not be as direct as we would like, but it may help people to visualise it.

For example, to demonstrate the electrical and magnetics of an atom, you could use the Atomic Viewer to show the structure of the element, then turn on the charge particles to show where it comes from and goes to (View Settings -> Proton Stack). With a few words and maybe some arrows imposed over the video, you could at least show the charge paths. It might be better than just hearing the words.

To demonstrate photon spins adding and canceling, you can use the Proton Charge Viewer. While not exactly showing what you want, it does provide something essentially the same. This app is from the perspective of a proton stack, where as you really want 2 separate protons and show their charge fields mingling. However, this app still can provide 2 protons (stacked but one is a proton and the next is an anti-proton) and their charge photons with axes on the photons to show their spins.

Have a look at a Firefox Plugin called Selenium. This allows you to record what you do in the browser and you can then 'play it back' so it does all of the same things again but without you having to do it correctly while recording. I don't know how to actually record it but there must be something out there to do it, I just haven't had the need to do so and haven't looked, but gamers do this kind of thing all the time.

Good luck. I hope it goes well.

I am adding a link to my Miles Mathis Mirror page as this thread comes up in a search for that.

www.nevyns-lab.com/mathis/mirror