Difference between revisions of "Roadmap"

From GnuCash
Jump to: navigation, search
(update some comments; make some statements somewhat stronger)
(Eliminate deprecated widgets and libraries: Update to Gtk4)
 
(20 intermediate revisions by 5 users not shown)
Line 1: Line 1:
This list represents is a rough list of ideas that some of the core developers have discussed over the years as the likely next directions for GnuCash.  That sentence is mightily-qualified as it should be; this list should not be interpreted as a promise or even really "official"... while it's a good guess as to what one group of people would like to do, projects like GnuCash are the results of a wide group of contributors, and they ultimately decide what road GnuCash travels.
+
= Overview =
 +
Below are the current major changes contemplated or in progress by the development team, roughly in priority order. Other proposed improvements of lower priority may be found on the [[WishList| Wish List]], but note that the development team believes that the work described here is a prerequisite to most everything found there.
 +
 
 +
= Replace GObject and GLib dependencies in the core =
 +
 
 +
Various parts of GnuCash use no objects, GObjects, and rather tortured emulation of objects in C. We have decided to reduce our dependence upon GLib, at least in the Engine and Backends, and to rewrite that part of the program in C++. In order to minimize external dependencies in that part of GnuCash, every effort will be made to depend only on the C++ Standard Library and [http://www.boost.org/ Boost]. We hope that this will make it easier to port GnuCash to other display platforms and GUI libraries.
 +
 
 +
More information, including some suggested reading if you're not fluent in C++, may be found on the [[C++]] page.
 +
 
 +
= Scheme minimization =
 +
 
 +
There are some parts of GnuCash that make a round-trip into scheme code for no really good reason.  The developers would like to throw out those parts of Scheme.
  
 
= Register rewrite =
 
= Register rewrite =
  
The current register is optimized for visual similarity to a "checkbook register" at the expense of simplicity and maintainability. It's also written at too low a level, in large part because it was written early in Gtk+-2's lifecycle when the appropriate layout widgets weren't available. The developers would love to see the register code being replaced by an implementation based on the GtkTreeView MVC model, but so far nobody has taken on this task again.
+
The current register code started out as code taken from the [http://www.gnumeric.org/ Gnumeric] spreadsheet design, greatly extended and customized for GnuCash's needs. This is central to GnuCash's look and feel with the basic view resembling a checkbook register while expanded views provide more detail about transactions.
 +
 
 +
It has been partially rewritten to use GtkLayout with a GtkEntry as active element. However the controller code is still mostly as it was. This doesn't map very well with Gtk's View-Controller philosophy. In particular, the code jumps through several hoops to map text entry between the GtkEntry and the controller code.
 +
 
 +
As a first step it would likely simplify a lot if the GtkEntry would be the sole responsible of all text entry (where now this is handled in large parts by gnucash-sheet, extended with control code from the abstract register implementation). Only keystrokes that can't be handled by the GtkEntry (like those for navigating around in a register) should be delegated to the underlying gnc-sheet.
 +
 
 +
A rewrite using GtkTreeModel was begun in 2.5 but wasn't completed. It is still in the code base and can be enabled during build configuration. While it addressed the obsolete library problem and would ease conversion to Gtk3, it is still tightly coupled to Gtk+ and so precludes building UIs with other libraries. Moreover as it's no longer maintained it's possibly no longer working and missing newer features that have been implemented in the current register in the meantime.
 +
 
 +
In addition, Gtk4 is moving away from GtkTrees in favour of GtkGrids. If a major rewrite is to be considered beyond GtkLayout and GtkEntry, it would probably be wise to evaluate the newer options Gtk4 has to offer.
  
 
= Eliminate deprecated widgets and libraries =
 
= Eliminate deprecated widgets and libraries =
  
Gnucash still uses several gtk and gnome widgets that have been marked as deprecated.  An effort is underway to eliminate most of these in the 2.2 release. There is one deprecated widget that can't be removed in that time frame because gtk is missing the equivalent functionality.
+
GnuCash still uses several gtk widgets that have been marked as deprecated, many of which are not supported in Gtk4. Update all of these to API shared between Gtk3 and Gtk4.
 +
 
 +
= Remove module system =
 +
 
 +
The current module system confuses runtime-loaded modules with shared librariesAs well, it is (arguably) at the wrong level of granularity in places, leading to too many modules. The developers would like to replace most of the plugins by one common library or static application, leaving only 2-3 really optional parts as runtime-loaded plugins (e.g. aqbanking).
 +
 
 +
= Model-View-Controller Pattern =
  
Several of the Gnome libraries upon which Gnucash now depends (libart_lgpl, libbonobo, libbonoboui, libglade, libgnome, libgnomecanvas, libgnomeprint, libgnomeprintui, libgnomeui, and libgnomevfs) have been deprecated for several years now, and will be '''''removed''''' from Gnome3, scheduled for release in January 2011. While most distributions will continue to support Gnome 2 for awhile, we'll need to migrate to the newer replacements soon.
+
While most of the "model" ''is'' properly encapsulated in src/engine, some has leaked into other parts of the program. Worse, there is little separation between view code and controller code. We need to separate the two to make it easier to implement GUIs with other GUI libraries.
  
In order to continue to provide backward compatbility, this may mean that we have to emulate Gtk+ and have two development branches, one for Gnome 2 and one for Gnome 3.
+
= Testing =
  
 +
Most subdirectories have a "test" directory containing tests of varying quality, but the coverage is poor and most of the tests are high-level tests that exercise the full module contained in the directory (for example, the dbi backend tests create a data structure in memory programmatically, then save it to a database and reload it and compare the reloaded data with the originals). These are good tests, but coverage is spotty. (Not all of the ways GnuCash stores data are covered in the dbi backend, for example.)
 +
 +
Modern practice calls for "unit tests" exercising every execution path in every exposed function in a tested module. Maintaining good unit tests during development has been extensively documented as a way to improve both productivity and quality. GnuCash doesn't have many unit tests, but it should.
 +
 +
Adding unit testing (with the [http://http://code.google.com/p/googletest/ Google Testing framework]) [[Testing |is in progress]]. Older tests written with GLib testing or the legacy test code in src/test-core will be converted to Google Test and extended further as part of the C++ rewrite.
  
 
= Database and QOF =
 
= Database and QOF =
Line 18: Line 48:
 
Now that we have the DBI backend supporting not just Sqlite3 but Postgresql and MySql as well, we want to move away from a "load everything into memory at the beginning" model to "load only what you need, and take out a backend write lock only on what you're editing" model to allow multiple simultaneous access, to speed up loading large datasets, better data integrity protection, and all of the other benefits provided by a good database.
 
Now that we have the DBI backend supporting not just Sqlite3 but Postgresql and MySql as well, we want to move away from a "load everything into memory at the beginning" model to "load only what you need, and take out a backend write lock only on what you're editing" model to allow multiple simultaneous access, to speed up loading large datasets, better data integrity protection, and all of the other benefits provided by a good database.
  
The current "Query Object Framework" that Gnucash uses rather gets in the way, because while it handles queries acceptably well, it doesn't offer any of the other services that a proper database does. So the "QOF" parts should rather be thrown out.
+
The current "Query Object Framework" that GnuCash uses rather gets in the way, because while it handles queries acceptably well, it doesn't offer any of the other services that a proper database does. So the "QOF" parts should rather be thrown out.
  
 
Some thoughts on what this requires:
 
Some thoughts on what this requires:
Line 26: Line 56:
 
* The set of splits for an account actually is modified with the running balance.  xaccAccountGetBalanceAsOfDate() uses this fact by updating the running balances and finding the first balance on or after the date.  Instead, the same db query used to get starting/ending balances should be used (sum all splits where account=desired account where date <= desired date).  The running balance should be kept by the register because it should be recomputed if the register is resorted or filtered (would close some bugs). (bug #591098 causes the "present" column on the account page to be 0 until the account register is opened if not all transactions are loaded at startup).
 
* The set of splits for an account actually is modified with the running balance.  xaccAccountGetBalanceAsOfDate() uses this fact by updating the running balances and finding the first balance on or after the date.  Instead, the same db query used to get starting/ending balances should be used (sum all splits where account=desired account where date <= desired date).  The running balance should be kept by the register because it should be recomputed if the register is resorted or filtered (would close some bugs). (bug #591098 causes the "present" column on the account page to be 0 until the account register is opened if not all transactions are loaded at startup).
  
= Scheme minimization =
+
====Proposal====
 
+
* Replace QofQuery with SQL queries to the SQL backend. XML files will be loaded as before, but instead of populating Engine objects it will be loaded into an in-memory SQLite3 database to moderated by the backend. Autosaves and end-of-session saves will be exports from the SQLite3 in-memory database to the XML file.
There are some parts of GnuCash that make a round-trip into scheme code for no really good reason. The developers would like to throw out those parts of Scheme.
+
* Non-editing GUI elements and reports will query the SQL backend instead of QofQuery. No Engine objects should be needed for queries. Non-editing GUI elements will take out read locks (so that multiple users can read the same database tuple). Only when a user edits should Engine objects be instantiated; when they are, the corresponding tuples in the database will be write (exclusive) locked. Note well that this will require careful deadlock management, because one cannot get a write lock on a read-locked tuple. Reports will run queries and will not use cursors or acquire any locks.
  
 
= Reports =
 
= Reports =
  
Right now reports are scheme scripts that programatically generate HTML.   
+
Right now reports are Scheme scripts that programmatically generate HTML.   
  
 
Scheme is impenetrable to most programmers. Expecting users to be able to write reports in Scheme is completely unreasonable.
 
Scheme is impenetrable to most programmers. Expecting users to be able to write reports in Scheme is completely unreasonable.
Line 40: Line 70:
 
* Record selection: Ideally graphical with an SQL-like "advanced" option
 
* Record selection: Ideally graphical with an SQL-like "advanced" option
 
* Layout: The original item here suggested an HTML template; that could be the "advanced" option, with a simple table/graph being the default
 
* Layout: The original item here suggested an HTML template; that could be the "advanced" option, with a simple table/graph being the default
* Style: CSS. It's built into WebKit, we should use it.
+
* Style: [[CSS]]. It's built into WebKit, we should use it.
* Javascript: WebKit supports javascript so complicated interactivity can be added.  Example: ability to expand/collapse levels of the account hierarchy in a report. This could be extended with Javascript interfaces to the API so that all of the report code is written in Javascript instead of Scheme.
+
* Javascript: WebKit supports Javascript so complicated interactivity can be added.  Example: ability to expand/collapse levels of the account hierarchy in a report. This could be extended with Javascript interfaces to the API so that all of the report code is written in Javascript instead of Scheme.
  
 
Any report module will still need some sort of scripting language to "calculate the numbers". Currently we have Scheme for this, but the developers would like to get away from that. Python might be a better option.
 
Any report module will still need some sort of scripting language to "calculate the numbers". Currently we have Scheme for this, but the developers would like to get away from that. Python might be a better option.
 
= Remove module system =
 
 
The current module system confuses runtime-loaded modules with shared libraries.  As well, it is (arguably) at the wrong level of granularity in places, leading to too many modules. The developers would like to replace most of the plugins by one common library or static application, leaving only 2-3 really optional parts as runtime-loaded plugins (e.g. aqbanking).
 
 
= Object Orientation and Language Selection =
 
 
Various parts of Gnucash use no objects, GObjects, and rather tortured emulation of objects in C. In the second decade of the 21st century, surely it's time to allow gradual replacement with proper object-oriented code written in an object-oriented language. We'll benefit with more readable, more portable, and more easily debuggable code that's also easier and faster to write. While C++ has the advantage that it integrates seamlessly with C and is available as a native language on all three targeted platforms, interpreted languages like Python and Ruby now perform well and are faster to write as well as having rich collections of adjunct libraries already ported to most operating systems which would greatly simplify oru cross-platform maintenance.  Many offer integration to multiple native GUIs (Gtk, Qt, Win32, .Net, and Cocoa) which would allow us to offer a better, more native, user experience on non-Linux platforms. The developers would like to see to move away from plain C and instead moving to one of the mentioned more modern languages.
 
 
= Testing =
 
 
Most subdirectories have a "test" directory containing tests of varying quality, but the coverage is poor and most of the tests are high-level tests that exercise the full module contained in the directory (for example, the dbi backend tests create a data structure in memory programmatically, then save it to a database and reload it and compare the reloaded data with the originals). These are good tests, but coverage is spotty. (Not all of the ways Gnucash stores data are covered in the dbi backend, for example.)
 
 
Modern practice calls for "unit tests" exercising every execution path in every exposed function in a tested module. Maintaining good unit tests during development has been extensively documented as a way to improve both productivity and quality. Gnucash doesn't have many unit tests, but it should.
 
 
Integrating testing into the build requires a good testing framework. A rudimentary one is provided as part of autotools, and Glib provides a perfectly acceptable unit test framework, gtest. Unit test frameworks are also available for all object-oriented languages that we're likely to adopt.
 

Latest revision as of 17:40, 18 December 2022

Overview

Below are the current major changes contemplated or in progress by the development team, roughly in priority order. Other proposed improvements of lower priority may be found on the Wish List, but note that the development team believes that the work described here is a prerequisite to most everything found there.

Replace GObject and GLib dependencies in the core

Various parts of GnuCash use no objects, GObjects, and rather tortured emulation of objects in C. We have decided to reduce our dependence upon GLib, at least in the Engine and Backends, and to rewrite that part of the program in C++. In order to minimize external dependencies in that part of GnuCash, every effort will be made to depend only on the C++ Standard Library and Boost. We hope that this will make it easier to port GnuCash to other display platforms and GUI libraries.

More information, including some suggested reading if you're not fluent in C++, may be found on the C++ page.

Scheme minimization

There are some parts of GnuCash that make a round-trip into scheme code for no really good reason. The developers would like to throw out those parts of Scheme.

Register rewrite

The current register code started out as code taken from the Gnumeric spreadsheet design, greatly extended and customized for GnuCash's needs. This is central to GnuCash's look and feel with the basic view resembling a checkbook register while expanded views provide more detail about transactions.

It has been partially rewritten to use GtkLayout with a GtkEntry as active element. However the controller code is still mostly as it was. This doesn't map very well with Gtk's View-Controller philosophy. In particular, the code jumps through several hoops to map text entry between the GtkEntry and the controller code.

As a first step it would likely simplify a lot if the GtkEntry would be the sole responsible of all text entry (where now this is handled in large parts by gnucash-sheet, extended with control code from the abstract register implementation). Only keystrokes that can't be handled by the GtkEntry (like those for navigating around in a register) should be delegated to the underlying gnc-sheet.

A rewrite using GtkTreeModel was begun in 2.5 but wasn't completed. It is still in the code base and can be enabled during build configuration. While it addressed the obsolete library problem and would ease conversion to Gtk3, it is still tightly coupled to Gtk+ and so precludes building UIs with other libraries. Moreover as it's no longer maintained it's possibly no longer working and missing newer features that have been implemented in the current register in the meantime.

In addition, Gtk4 is moving away from GtkTrees in favour of GtkGrids. If a major rewrite is to be considered beyond GtkLayout and GtkEntry, it would probably be wise to evaluate the newer options Gtk4 has to offer.

Eliminate deprecated widgets and libraries

GnuCash still uses several gtk widgets that have been marked as deprecated, many of which are not supported in Gtk4. Update all of these to API shared between Gtk3 and Gtk4.

Remove module system

The current module system confuses runtime-loaded modules with shared libraries. As well, it is (arguably) at the wrong level of granularity in places, leading to too many modules. The developers would like to replace most of the plugins by one common library or static application, leaving only 2-3 really optional parts as runtime-loaded plugins (e.g. aqbanking).

Model-View-Controller Pattern

While most of the "model" is properly encapsulated in src/engine, some has leaked into other parts of the program. Worse, there is little separation between view code and controller code. We need to separate the two to make it easier to implement GUIs with other GUI libraries.

Testing

Most subdirectories have a "test" directory containing tests of varying quality, but the coverage is poor and most of the tests are high-level tests that exercise the full module contained in the directory (for example, the dbi backend tests create a data structure in memory programmatically, then save it to a database and reload it and compare the reloaded data with the originals). These are good tests, but coverage is spotty. (Not all of the ways GnuCash stores data are covered in the dbi backend, for example.)

Modern practice calls for "unit tests" exercising every execution path in every exposed function in a tested module. Maintaining good unit tests during development has been extensively documented as a way to improve both productivity and quality. GnuCash doesn't have many unit tests, but it should.

Adding unit testing (with the Google Testing framework) is in progress. Older tests written with GLib testing or the legacy test code in src/test-core will be converted to Google Test and extended further as part of the C++ rewrite.

Database and QOF

Now that we have the DBI backend supporting not just Sqlite3 but Postgresql and MySql as well, we want to move away from a "load everything into memory at the beginning" model to "load only what you need, and take out a backend write lock only on what you're editing" model to allow multiple simultaneous access, to speed up loading large datasets, better data integrity protection, and all of the other benefits provided by a good database.

The current "Query Object Framework" that GnuCash uses rather gets in the way, because while it handles queries acceptably well, it doesn't offer any of the other services that a proper database does. So the "QOF" parts should rather be thrown out.

Some thoughts on what this requires:

  • Transactions and splits are by far the object types which cause the most problem.
  • Some other objects contain a pointer to a transaction or split (e.g. an invoice contains the pointer to the posting transaction). Either the transaction needs to be loaded when the invoice is or else it needs to be loaded when it is first referenced.
  • Some reports generate values by creating a query on splits and then summing them. Given a database engine, a report should be able to query the value of splits for an account or set of accounts on a specific date (with possibly other qualifiers as well e.g. reconcile status). A database engine can do this natively, C code could do it while we're converting, but the API should be defined.
  • The set of splits for an account actually is modified with the running balance. xaccAccountGetBalanceAsOfDate() uses this fact by updating the running balances and finding the first balance on or after the date. Instead, the same db query used to get starting/ending balances should be used (sum all splits where account=desired account where date <= desired date). The running balance should be kept by the register because it should be recomputed if the register is resorted or filtered (would close some bugs). (bug #591098 causes the "present" column on the account page to be 0 until the account register is opened if not all transactions are loaded at startup).

Proposal

  • Replace QofQuery with SQL queries to the SQL backend. XML files will be loaded as before, but instead of populating Engine objects it will be loaded into an in-memory SQLite3 database to moderated by the backend. Autosaves and end-of-session saves will be exports from the SQLite3 in-memory database to the XML file.
  • Non-editing GUI elements and reports will query the SQL backend instead of QofQuery. No Engine objects should be needed for queries. Non-editing GUI elements will take out read locks (so that multiple users can read the same database tuple). Only when a user edits should Engine objects be instantiated; when they are, the corresponding tuples in the database will be write (exclusive) locked. Note well that this will require careful deadlock management, because one cannot get a write lock on a read-locked tuple. Reports will run queries and will not use cursors or acquire any locks.

Reports

Right now reports are Scheme scripts that programmatically generate HTML.

Scheme is impenetrable to most programmers. Expecting users to be able to write reports in Scheme is completely unreasonable.

The ideal solution should have three modules:

  • Record selection: Ideally graphical with an SQL-like "advanced" option
  • Layout: The original item here suggested an HTML template; that could be the "advanced" option, with a simple table/graph being the default
  • Style: CSS. It's built into WebKit, we should use it.
  • Javascript: WebKit supports Javascript so complicated interactivity can be added. Example: ability to expand/collapse levels of the account hierarchy in a report. This could be extended with Javascript interfaces to the API so that all of the report code is written in Javascript instead of Scheme.

Any report module will still need some sort of scripting language to "calculate the numbers". Currently we have Scheme for this, but the developers would like to get away from that. Python might be a better option.