[Discussion] Docs Repo restructuring

[Confectrician]

Hi all,

One thing that i could observe repeatedly since i am helping out here in the docs
is the confusion about external content.
I have added the external source label some time ago for this and i am using it frequently.

The Problem

People come here to improve the docs and me or others have to tell them that they are in the wrong place.
This is one thing if its only a new issue.
Its a bigger problem if someone already invested time, prepared an improvement and opened a PR.

This is especially hard for new contributors who have to get around

• with the docs itself,
• with jekyll
  (which has a really flat learning curve and big entry hurdle, to be honest)
• with our big/grown repo structure
• with a difficult system of generated files from external sources

The result is that some of them give up fast or even don't start thinking about contributing here.
And i can understand that sometimes to be honest.

First of all you will have to get jekyll running on your machine.
I first tried the vagrant machine, which had problem with refreshing after a file save.
(I think there was an issue about it.)
After being annoyed of that after some time i switched to the ruby variant.
It works but you have to do many steps until you can do what you originally wanted.
Work on those documentation.

Some time later, to me happened the same that happened to many others.

• I had made an improvement (i think it was about some transformation)
• I overread the "Please edit the original file"- Comment
• I had to move it to eclipse smarthome later

Fun Fact:
We discussed this change with @ThomDietrich before and even he overread that sentence in this case.

For me it was no big deal, because i already provided some commits to the contribution and was already a bit familiar with it.
But i can imagine that this can be frustrating for someone with no experience here at all.

What is my Resumee now

We are making contributing for the docs way to hard especially for unexpierienced People.
I could imagine that docs would be a perfect place for people who are not familiar with programming,
but with a will to contribute to the Project in any way.

How can we Improve this?

This is the Topic i want to discuss here.
But maybe i will go a step back first and would ask you for your opinion:

• Do you even agree with my observations?
• Do you also think the docs are confusing with all its external content and structure?
• Do you maybe see some additional problems?

Maybe some ideas for starting with improvements

Now i have complained for many lines.
Time to give something back in form of a possible improvement

A first suggestion

My suggestion would be to change the Repo structure a little bit.

1. All final contents of the docs page in a new repo which is filled only through generated contents
   from the different sources. (Bindings, Docs-Articles, ESH, ...)
   Nothing should be changed here by Pull Request.
2. A pure-docs repo where only Documentation Articles can be found.

1. would then be the source for our homepage generation (jekyll or whatever the future will bring)
2. would be the base for contributors to the docs.

Which benefits do you expect with this?

The idea is simple:
If there isn't a file (like a binding readme) to edit in the repo, people will (hopefully) search for it or ask where to find it.
This way we can inform and point to the right direction before someone has made a big effort in the wrong place.

It also reduces the amount of files and folders in the docs repository which hopefully will also help first contributors with starting here.

A second suggestion

We should rework/improve existing guides for the folder structure and generation of external contents.
It should be easier to find out where to look for a specific content/file/...

A third suggestion

This is more a general one which could be thought of for all repos.
I think i have seen this approach in the nodejs repo.
They offer some kind of mentoring with an mentor available label for their issues.

This could be a good thing for new contributors and of course this is not limited to the documentation.

Thanks for reading

Many line from my side now.
I was thinking about this topic(s) some time already and now thought it would be good to write it down here for discussion.
Now it's your part and i hope to get some feedback collected here about this topic.

I would love to add additionally suggestions from your side to the first Post. 😃

BR
Jerome

[bgilmer77]

I am at a very busy trade show so I must keep this short. I agree that there is a problem. I also see many times we have to say, “thank you for your contribution. Now please do your work again, but somewhere else”. I can imagine this is very frustrating. I do not have time to look at the proposed solutions, but thanks for highlighting the problem.

[ThomDietrich]

Hey!
This is an annoying and unneeded problem indeed. I want to quickly give another view on the whole issue.

The actual source if this issue is, that we copy external content over to this repository. This is only needed because we are hosting the documentation via GitHub. One limitation that emerges out of that is, that we indeed need to commit all documents. This is also the reason for the regular "Generated Content" commits and the precompiled "v2.1.0" folder. All of that could be easily avoided if we were to serve the documentation on a system hosted by ourselves. @kaikreuzer had a few arguments against that, the main one being the additional administrative stress on a single person - which is of course a serious concern in an open source community. Driving this idea forward would be one of my goals for after I'm back in the game (which is btw around September of this year).

In the mean time I applaud your urge to find a solution here and now. I'm not so sure about your second suggestion and didn't understand the third. The first however (if you think about it) already goes in the direction of what I've described above. Your final repo would be the result of what should automatically be generated on the self-hosted server. I've already improved the external sources scripts to automate that process, surely it can still be improved. That would be needed before we can self-host a self-updating version of the documentation. Therefore your final repo idea (and according implementations) would be a good step into this direction.

In fact the final repo idea could be a long-lasting alternative to my self-hosting approach. Because of that you have my full "spiritual" (sadly no practical) support!

[Confectrician]

I'm not so sure about your second suggestion

Let me explain it in some simple words:
I am helping out here for a not so short time now and i am still not aware of the whole generation process.
I also think that is really unlucky to have to reverse engineer the generation process from xml or shell files.
It should be easy to grasp for everyone which contents are coming from which source/repo.
That should be improved with a graphic or better explanation.

Currently that's only a sub-part of the readme.
I think that could be presented more popular since the repo structure and external sources like esh are a fundamental part.

[kaikreuzer]

Hey @Confectrician, I agree with the problem you are raising and as many things, this has grown historically to what it is today.

I think a separation of "source docs" and "served pages" indeed makes a lot of sense - mixing those simply causes a lot of mess. Maybe there is actually a not too complex short term solution for it without requiring additional repos or servers:

We could remove all stuff that is NOT meant for manual editing from the master branch of openhab-docs and set up a Jenkins job, which runs the generation and copies everything to a gh-pages branch in the same repo, from which Github then serves the website. This way everything would be hidden in a separate branch and would not cause any confusion on master.

Note that this is only my idea for the short term. Mid-term (and that might fit to @ThomDietrich's September timeline, btw, good to hear that you plan to be back!) I am currently preparing a website relaunch, which will also have to take the documentation into account. Nothing is decided yet, but one of the ideas was to use a framework like NUXT instead of Jekyll - this could be used in the start to generate static pages from the sources that are then pushed to Github Pages, but it would also offer us the possibility to run it dynamically on a server, so that we could eventually end up with @ThomDietrich's dream of a self-hosted page :-)

[bgilmer77]

All,
Again, very abbreviated comments, but I would like to contribute to this discussion.
A few points:

• We should always think very carefully about anything that makes it difficult for people to contribute. Some people who put a contribution in the wrong repo will be okay to move it. Others will go away and never come back if we give them the "fail - put your contribution over here" message. That is not good for building a community over the long haul. No one likes doing documentation, so anyone we get who contributes should have the easiest path possible.

• Adding infrastructure which requires an ongoing commitment to maintenance needs to be considered very carefully. In an open source community, people come and go. If a dedicated hosting situation relies on an individual, at some point, that hosting will encounter a people problem. The project could suffer very badly.

Of course, both of these points are important, and both are conflicting. So having a good discussion and considering carefully before making changes is a very good idea. Therefore, I am really happy to see the discussion here.

Finally, I would point out that the documentation in OH, in particular, is critical. If you look at the Community, you see many cases where people say, "I worked for days - I give up". I expect that this is just a fact of life. But what it means is that we always need to work to make the documentation as good as it can be in order to have the project be successful.

Thank you
Brad

[ThomDietrich]

@kaikreuzer your short term solution is pretty similar to @Confectrician 's idea to add a new repo but even better. @Confectrician I think it's totally worth trying. What we would need is a bash script (at least that is one and maybe the most logical choice) that executes all external-sources-steps and adds the appropriate git steps affecting the gh-pages branch.

@kaikreuzer I'm amazed that we didn't think about this "second repo/branch" idea yet. Kudos to @Confectrician

@kaikreuzer I don't want to start this discussion here and now but I would be sad to abandon Jekyll. It has been a reliable and capable companion so far and the problems we wanted to solve with the webpage relaunch were not the fault of Jekyll but of missing implementation work.

@kaikreuzer September should be realistic. I'll be back eventually, no doubt about that!

[Confectrician]

Hi all,

First i want to thank all of you for the huge and constructive feedback in that short time.

The branch solution sounds good for me too. (For a short term improvement)
Keep it short and simple on master to avoid confusion for new contributors and keep the complicated stuff (which has currently to be here due to the GitHub Pages workflow, which i fully understand) "under the hood" for the first view.

I am currently preparing a website relaunch, which will also have to take the documentation into account.

I am aware of that and the corresponding community topic, but wanted to start this discussion on an early stage.
I think docs will be the biggest part (amount of content as well as work to do) of that relaunch and it would be good to get some feedback and opinions as early as possible.

[bgilmer77]

FWIW, as a relative newcomer, I liked Jekyll.

[Confectrician]

What we would need is a bash script (at least that is one and maybe the most logical choice) that executes all external-sources-steps and adds the appropriate git steps affecting the gh-pages branch.

I would use the current external content workflow as is and just push that to the hidden branch instead of master in the future.
It doesn't make sense to invest much time in automating this stuff so short before a possible website relaunch.

For the docs-master itself i would love to test/implement some automation already.

Workflow could be

• master commit
  -> update gh-pages branch due CI/CD job

• commit to (hidden) external content branch from our exesting generation
  -> update gh-pages branch due CI/CD job

CI/CD

If i can i would love to get some introduction here and help out maintaining/obesrving this.
I have no real experience with CI, just had a quick research at travis some time ago, while informing for the docs or related to the website relaunch.

Possible To-Do

Before i am going on:
I am currently working for my master degree, so pls be aware that my spare time might be not that much during some timespans. I think you may be in a similar situation @ThomDietrich

• Determine the indiviual contents of each branch
• Prepare all branches while leaving the master branch as is (temporary),
  to keep the docs page available during the whole time
• Set up CI job, clean master branch, change repo settings to use the gh-pages branch

Did I miss something?

[Confectrician]

Hey @ghys,

As you may have already recognised i am a bit busy these days. (Finishing my master degree in july, so that has more priority atm.)

Since you made huge progress for the website relaunch in the last time, maybe you could share your opinion here too.

What would be the best design to make generating the docs into the new page by script as easy as possible in the future.

I would also be interested in digging into the vue js stuff after finishing my master degree and help you out there.

Until that point i will have to be less active here, but i will try to follow up on the issues and PRs if possible.

[ghys]

This is something I've been having on my mind but still have to find a good solution for. I'll think about it a little bit and make some proposals here soon (I hope!)

[ghys]

I have created #700 but to clarify about my intent, it's only the very first step. For now it's only a way to keep the generated content regularly updated.
A second step would be to implement what was proposed here: #665 (comment)

i.e. switching the default branch to "master", with only the non-generated documentation content, and have this build process maintain the current gh-pages branch (not presented by default to users) which would hold the contents from master + the generated content. The new website will fetch content from this branch as it does now.

[Confectrician]

I needed a pause from my master degree work,
so i have introduced a master branch to my personal fork of the docs repo
to get a bit progress here.

https://github.com/confectrician/openhab-docs/tree/master

Maybe you can have a look at it.
This is of course wip.
If there are any files that shouldn't be on the later master branch from your perspective
leave a comment.

When the branch is final i will push it to this repo too.

[Confectrician]

I think we could remove imprint, privacy. since this is handled global now.

What about pom.xml, process_addons.groovy and update_external_ressources.sh?
Should they stay at master for now until we have set up an automated process?

[ghys]

I would remove the Jekyll-related stuff like:

• _layouts
• _includes
• _config.yml
• Vagrantfile
  and probably 404.md too since it's not used on the website.

[ghys]

What about pom.xml, process_addons.groovy and update_external_ressources.sh?
Should they stay at master for now until we have set up an automated process?

Maybe it's best to leave them for now since we don't know which branch the build script will work with.

[Confectrician]

Pleasure to close this one.
I like the clean new structure and i hope it will make some interested users to interested contributors in the nearer future. :)